2024-04-24T14:20:06.887 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T14:20:06.887 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi --owner scheduled_yuriw@teuthology --verbose --description rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} thrashers/cache thrashosds-health workloads/rbd_nbd} --archive /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510 -- /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510/orig.config.yaml 2024-04-24T14:20:06.924 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T14:20:07.012 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510 branch: wip-yuriw4-testing-20240423.151325-quincy description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} thrashers/cache thrashosds-health workloads/rbd_nbd} email: yweinste@redhat.com first_in_suite: false job_id: '7671510' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuriw4-testing-20240423.151325-quincy ceph: conf: global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(CACHE_POOL_NEAR_FULL\) - \(CACHE_POOL_NO_HIT_SET\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: extra_packages: - rbd-nbd flavor: default sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuriw4-testing-20240423.151325-quincy sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - - client.0 seed: 9954 sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-yuriw4-testing-20240423.151325-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 targets: smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create cache 4 - sudo ceph osd tier add rbd cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay rbd cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 60 - sudo ceph osd pool set cache target_max_objects 250 - thrashosds: timeout: 1200 - workunit: clients: client.0: - rbd/rbd-nbd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_14:09:24 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T14:20:07.012 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa; will attempt to use it 2024-04-24T14:20:07.013 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa/tasks 2024-04-24T14:20:07.013 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T14:20:07.015 INFO:teuthology.task.internal:Checking packages... 2024-04-24T14:20:07.040 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '330067dbce47655d1f762ef14d546d2fde16a6a9' 2024-04-24T14:20:07.040 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T14:20:07.040 INFO:teuthology.packaging:ref: None 2024-04-24T14:20:07.041 INFO:teuthology.packaging:tag: None 2024-04-24T14:20:07.041 INFO:teuthology.packaging:branch: wip-yuriw4-testing-20240423.151325-quincy 2024-04-24T14:20:07.041 INFO:teuthology.packaging:sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T14:20:07.041 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuriw4-testing-20240423.151325-quincy 2024-04-24T14:20:07.188 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-915-g330067db-1focal 2024-04-24T14:20:07.190 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T14:20:07.196 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T14:20:07.197 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T14:20:07.221 INFO:teuthology.task.internal:Saving configuration 2024-04-24T14:20:07.235 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T14:20:07.255 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T14:20:07.282 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 14:12:59.383471', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-04-24T14:20:07.303 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 14:12:59.384933', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-04-24T14:20:07.324 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 14:12:59.386067', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-04-24T14:20:07.324 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T14:20:07.330 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-04-24T14:20:07.330 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'] 2024-04-24T14:20:07.330 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['client.0'] 2024-04-24T14:20:07.330 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T14:20:07.449 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fda9ca23f70>, signals=[15]) 2024-04-24T14:20:07.450 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T14:20:07.457 INFO:teuthology.task.internal:Opening connections... 2024-04-24T14:20:07.457 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-04-24T14:20:07.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:20:07.548 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T14:20:07.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:20:07.637 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-04-24T14:20:07.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:20:07.726 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T14:20:07.734 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-04-24T14:20:07.741 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-04-24T14:20:07.742 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-04-24T14:20:07.790 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="Ubuntu" 2024-04-24T14:20:07.790 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-04-24T14:20:07.790 INFO:teuthology.orchestra.run.smithi090.stdout:ID=ubuntu 2024-04-24T14:20:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE=debian 2024-04-24T14:20:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-04-24T14:20:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="20.04" 2024-04-24T14:20:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-24T14:20:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-24T14:20:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-24T14:20:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-24T14:20:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_CODENAME=focal 2024-04-24T14:20:07.791 INFO:teuthology.orchestra.run.smithi090.stdout:UBUNTU_CODENAME=focal 2024-04-24T14:20:07.792 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-04-24T14:20:07.822 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-04-24T14:20:07.829 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-04-24T14:20:07.829 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-04-24T14:20:07.880 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="Ubuntu" 2024-04-24T14:20:07.880 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-04-24T14:20:07.880 INFO:teuthology.orchestra.run.smithi103.stdout:ID=ubuntu 2024-04-24T14:20:07.880 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE=debian 2024-04-24T14:20:07.880 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-04-24T14:20:07.880 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="20.04" 2024-04-24T14:20:07.880 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-24T14:20:07.881 INFO:teuthology.orchestra.run.smithi103.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-24T14:20:07.881 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-24T14:20:07.881 INFO:teuthology.orchestra.run.smithi103.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-24T14:20:07.881 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_CODENAME=focal 2024-04-24T14:20:07.881 INFO:teuthology.orchestra.run.smithi103.stdout:UBUNTU_CODENAME=focal 2024-04-24T14:20:07.881 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-04-24T14:20:07.905 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-04-24T14:20:07.912 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-04-24T14:20:07.912 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-04-24T14:20:07.960 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="Ubuntu" 2024-04-24T14:20:07.960 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-04-24T14:20:07.961 INFO:teuthology.orchestra.run.smithi173.stdout:ID=ubuntu 2024-04-24T14:20:07.961 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE=debian 2024-04-24T14:20:07.961 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-04-24T14:20:07.961 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="20.04" 2024-04-24T14:20:07.961 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-24T14:20:07.961 INFO:teuthology.orchestra.run.smithi173.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-24T14:20:07.961 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-24T14:20:07.961 INFO:teuthology.orchestra.run.smithi173.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-24T14:20:07.961 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_CODENAME=focal 2024-04-24T14:20:07.961 INFO:teuthology.orchestra.run.smithi173.stdout:UBUNTU_CODENAME=focal 2024-04-24T14:20:07.962 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-04-24T14:20:07.980 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T14:20:07.989 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T14:20:08.021 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T14:20:08.022 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-04-24T14:20:08.024 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-04-24T14:20:08.026 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-04-24T14:20:08.032 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T14:20:08.046 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T14:20:08.047 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-04-24T14:20:08.070 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-04-24T14:20:08.072 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-04-24T14:20:08.084 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T14:20:08.164 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T14:20:08.180 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-24T14:20:08.180 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'client.0': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-24T14:20:08.181 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T14:20:08.181 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T14:20:08.182 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T14:20:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T14:20:08.187 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-04-24T14:20:08.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T14:20:08.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T14:20:08.189 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-04-24T14:20:08.189 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-04-24T14:20:08.198 INFO:teuthology.orchestra.run.smithi103.stdout:5.4.0-124-generic 2024-04-24T14:20:08.198 INFO:teuthology.task.kernel:Running kernel on smithi103: 5.4.0-124-generic 2024-04-24T14:20:08.199 DEBUG:teuthology.orchestra.run.smithi103:> sudo apt-get clean 2024-04-24T14:20:08.200 INFO:teuthology.orchestra.run.smithi173.stdout:5.4.0-124-generic 2024-04-24T14:20:08.200 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.4.0-124-generic 2024-04-24T14:20:08.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-get clean 2024-04-24T14:20:08.238 INFO:teuthology.orchestra.run.smithi090.stdout:5.4.0-124-generic 2024-04-24T14:20:08.238 INFO:teuthology.task.kernel:Running kernel on smithi090: 5.4.0-124-generic 2024-04-24T14:20:08.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get clean 2024-04-24T14:20:08.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-get update 2024-04-24T14:20:08.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get update 2024-04-24T14:20:08.339 DEBUG:teuthology.orchestra.run.smithi103:> sudo apt-get update 2024-04-24T14:20:08.486 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-04-24T14:20:08.501 INFO:teuthology.orchestra.run.smithi173.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2024-04-24T14:20:08.501 INFO:teuthology.orchestra.run.smithi173.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2024-04-24T14:20:08.508 INFO:teuthology.orchestra.run.smithi090.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2024-04-24T14:20:08.541 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-04-24T14:20:08.541 INFO:teuthology.orchestra.run.smithi103.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2024-04-24T14:20:08.541 INFO:teuthology.orchestra.run.smithi103.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2024-04-24T14:20:08.578 INFO:teuthology.orchestra.run.smithi173.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2024-04-24T14:20:08.594 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-04-24T14:20:08.598 INFO:teuthology.orchestra.run.smithi103.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2024-04-24T14:20:08.676 INFO:teuthology.orchestra.run.smithi090.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2024-04-24T14:20:08.833 INFO:teuthology.orchestra.run.smithi090.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,889 kB] 2024-04-24T14:20:08.926 INFO:teuthology.orchestra.run.smithi173.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,266 kB] 2024-04-24T14:20:08.933 INFO:teuthology.orchestra.run.smithi103.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [743 kB] 2024-04-24T14:20:09.028 INFO:teuthology.orchestra.run.smithi173.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [743 kB] 2024-04-24T14:20:09.033 INFO:teuthology.orchestra.run.smithi090.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2024-04-24T14:20:09.061 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [743 kB] 2024-04-24T14:20:09.072 INFO:teuthology.orchestra.run.smithi090.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [435 kB] 2024-04-24T14:20:09.081 INFO:teuthology.orchestra.run.smithi090.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2024-04-24T14:20:09.082 INFO:teuthology.orchestra.run.smithi090.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,772 kB] 2024-04-24T14:20:09.090 INFO:teuthology.orchestra.run.smithi103.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,266 kB] 2024-04-24T14:20:09.092 INFO:teuthology.orchestra.run.smithi103.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,889 kB] 2024-04-24T14:20:09.143 INFO:teuthology.orchestra.run.smithi090.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [36.4 kB] 2024-04-24T14:20:09.144 INFO:teuthology.orchestra.run.smithi090.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [387 kB] 2024-04-24T14:20:09.152 INFO:teuthology.orchestra.run.smithi090.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2024-04-24T14:20:09.152 INFO:teuthology.orchestra.run.smithi090.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [656 kB] 2024-04-24T14:20:09.165 INFO:teuthology.orchestra.run.smithi090.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [967 kB] 2024-04-24T14:20:09.202 INFO:teuthology.orchestra.run.smithi090.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [956 kB] 2024-04-24T14:20:09.239 INFO:teuthology.orchestra.run.smithi090.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [201 kB] 2024-04-24T14:20:09.244 INFO:teuthology.orchestra.run.smithi090.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-04-24T14:20:09.244 INFO:teuthology.orchestra.run.smithi090.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,200 B] 2024-04-24T14:20:09.245 INFO:teuthology.orchestra.run.smithi090.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.0 kB] 2024-04-24T14:20:09.245 INFO:teuthology.orchestra.run.smithi090.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,904 B] 2024-04-24T14:20:09.246 INFO:teuthology.orchestra.run.smithi090.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2024-04-24T14:20:09.258 INFO:teuthology.orchestra.run.smithi103.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [435 kB] 2024-04-24T14:20:09.275 INFO:teuthology.orchestra.run.smithi103.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2024-04-24T14:20:09.275 INFO:teuthology.orchestra.run.smithi103.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,772 kB] 2024-04-24T14:20:09.301 INFO:teuthology.orchestra.run.smithi173.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,889 kB] 2024-04-24T14:20:09.337 INFO:teuthology.orchestra.run.smithi103.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [36.4 kB] 2024-04-24T14:20:09.338 INFO:teuthology.orchestra.run.smithi103.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [387 kB] 2024-04-24T14:20:09.344 INFO:teuthology.orchestra.run.smithi103.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2024-04-24T14:20:09.344 INFO:teuthology.orchestra.run.smithi103.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [656 kB] 2024-04-24T14:20:09.354 INFO:teuthology.orchestra.run.smithi103.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [956 kB] 2024-04-24T14:20:09.372 INFO:teuthology.orchestra.run.smithi103.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [201 kB] 2024-04-24T14:20:09.372 INFO:teuthology.orchestra.run.smithi173.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [967 kB] 2024-04-24T14:20:09.380 INFO:teuthology.orchestra.run.smithi103.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-04-24T14:20:09.380 INFO:teuthology.orchestra.run.smithi103.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,200 B] 2024-04-24T14:20:09.380 INFO:teuthology.orchestra.run.smithi103.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.0 kB] 2024-04-24T14:20:09.391 INFO:teuthology.orchestra.run.smithi103.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,904 B] 2024-04-24T14:20:09.392 INFO:teuthology.orchestra.run.smithi103.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2024-04-24T14:20:09.422 INFO:teuthology.orchestra.run.smithi090.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,266 kB] 2024-04-24T14:20:09.428 INFO:teuthology.orchestra.run.smithi173.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [517 kB] 2024-04-24T14:20:09.443 INFO:teuthology.orchestra.run.smithi103.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [967 kB] 2024-04-24T14:20:09.456 INFO:teuthology.orchestra.run.smithi103.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [517 kB] 2024-04-24T14:20:09.469 INFO:teuthology.orchestra.run.smithi103.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2024-04-24T14:20:09.469 INFO:teuthology.orchestra.run.smithi103.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,889 kB] 2024-04-24T14:20:09.472 INFO:teuthology.orchestra.run.smithi173.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2024-04-24T14:20:09.512 INFO:teuthology.orchestra.run.smithi173.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [37.7 kB] 2024-04-24T14:20:09.531 INFO:teuthology.orchestra.run.smithi103.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [37.7 kB] 2024-04-24T14:20:09.531 INFO:teuthology.orchestra.run.smithi103.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [403 kB] 2024-04-24T14:20:09.537 INFO:teuthology.orchestra.run.smithi103.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2024-04-24T14:20:09.537 INFO:teuthology.orchestra.run.smithi103.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,181 kB] 2024-04-24T14:20:09.560 INFO:teuthology.orchestra.run.smithi103.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [783 kB] 2024-04-24T14:20:09.568 INFO:teuthology.orchestra.run.smithi173.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,889 kB] 2024-04-24T14:20:09.576 INFO:teuthology.orchestra.run.smithi103.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [283 kB] 2024-04-24T14:20:09.585 INFO:teuthology.orchestra.run.smithi103.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2024-04-24T14:20:09.589 INFO:teuthology.orchestra.run.smithi090.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [517 kB] 2024-04-24T14:20:09.601 INFO:teuthology.orchestra.run.smithi090.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2024-04-24T14:20:09.601 INFO:teuthology.orchestra.run.smithi090.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [37.7 kB] 2024-04-24T14:20:09.602 INFO:teuthology.orchestra.run.smithi090.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,889 kB] 2024-04-24T14:20:09.644 INFO:teuthology.orchestra.run.smithi103.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [26.2 kB] 2024-04-24T14:20:09.645 INFO:teuthology.orchestra.run.smithi103.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,444 B] 2024-04-24T14:20:09.646 INFO:teuthology.orchestra.run.smithi103.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,880 B] 2024-04-24T14:20:09.646 INFO:teuthology.orchestra.run.smithi103.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2024-04-24T14:20:09.646 INFO:teuthology.orchestra.run.smithi103.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-04-24T14:20:09.647 INFO:teuthology.orchestra.run.smithi103.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-04-24T14:20:09.648 INFO:teuthology.orchestra.run.smithi103.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-04-24T14:20:09.649 INFO:teuthology.orchestra.run.smithi103.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-04-24T14:20:09.649 INFO:teuthology.orchestra.run.smithi103.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-04-24T14:20:09.668 INFO:teuthology.orchestra.run.smithi103.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-04-24T14:20:09.675 INFO:teuthology.orchestra.run.smithi173.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [403 kB] 2024-04-24T14:20:09.680 INFO:teuthology.orchestra.run.smithi090.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [403 kB] 2024-04-24T14:20:09.687 INFO:teuthology.orchestra.run.smithi090.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2024-04-24T14:20:09.703 INFO:teuthology.orchestra.run.smithi173.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2024-04-24T14:20:09.724 INFO:teuthology.orchestra.run.smithi173.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,181 kB] 2024-04-24T14:20:09.768 INFO:teuthology.orchestra.run.smithi173.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [783 kB] 2024-04-24T14:20:09.803 INFO:teuthology.orchestra.run.smithi173.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [283 kB] 2024-04-24T14:20:09.814 INFO:teuthology.orchestra.run.smithi173.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [435 kB] 2024-04-24T14:20:09.828 INFO:teuthology.orchestra.run.smithi173.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2024-04-24T14:20:09.850 INFO:teuthology.orchestra.run.smithi173.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,444 B] 2024-04-24T14:20:09.851 INFO:teuthology.orchestra.run.smithi090.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [783 kB] 2024-04-24T14:20:09.866 INFO:teuthology.orchestra.run.smithi173.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2024-04-24T14:20:09.867 INFO:teuthology.orchestra.run.smithi173.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,772 kB] 2024-04-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi173.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [26.2 kB] 2024-04-24T14:20:09.894 INFO:teuthology.orchestra.run.smithi173.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,880 B] 2024-04-24T14:20:09.980 INFO:teuthology.orchestra.run.smithi173.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2024-04-24T14:20:10.002 INFO:teuthology.orchestra.run.smithi173.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-04-24T14:20:10.024 INFO:teuthology.orchestra.run.smithi173.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-04-24T14:20:10.047 INFO:teuthology.orchestra.run.smithi173.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-04-24T14:20:10.069 INFO:teuthology.orchestra.run.smithi173.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-04-24T14:20:10.079 INFO:teuthology.orchestra.run.smithi173.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [36.4 kB] 2024-04-24T14:20:10.081 INFO:teuthology.orchestra.run.smithi173.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [387 kB] 2024-04-24T14:20:10.091 INFO:teuthology.orchestra.run.smithi173.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-04-24T14:20:10.106 INFO:teuthology.orchestra.run.smithi173.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2024-04-24T14:20:10.106 INFO:teuthology.orchestra.run.smithi173.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [656 kB] 2024-04-24T14:20:10.113 INFO:teuthology.orchestra.run.smithi173.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-04-24T14:20:10.147 INFO:teuthology.orchestra.run.smithi173.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [956 kB] 2024-04-24T14:20:10.203 INFO:teuthology.orchestra.run.smithi173.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [201 kB] 2024-04-24T14:20:10.214 INFO:teuthology.orchestra.run.smithi173.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-04-24T14:20:10.214 INFO:teuthology.orchestra.run.smithi173.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.0 kB] 2024-04-24T14:20:10.215 INFO:teuthology.orchestra.run.smithi173.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,200 B] 2024-04-24T14:20:10.215 INFO:teuthology.orchestra.run.smithi173.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,904 B] 2024-04-24T14:20:10.215 INFO:teuthology.orchestra.run.smithi173.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2024-04-24T14:20:10.442 INFO:teuthology.orchestra.run.smithi090.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,181 kB] 2024-04-24T14:20:10.551 INFO:teuthology.orchestra.run.smithi090.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [283 kB] 2024-04-24T14:20:10.572 INFO:teuthology.orchestra.run.smithi090.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2024-04-24T14:20:10.573 INFO:teuthology.orchestra.run.smithi090.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,444 B] 2024-04-24T14:20:10.575 INFO:teuthology.orchestra.run.smithi090.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [26.2 kB] 2024-04-24T14:20:10.576 INFO:teuthology.orchestra.run.smithi090.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,880 B] 2024-04-24T14:20:10.577 INFO:teuthology.orchestra.run.smithi090.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2024-04-24T14:20:10.577 INFO:teuthology.orchestra.run.smithi090.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-04-24T14:20:10.580 INFO:teuthology.orchestra.run.smithi090.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-04-24T14:20:10.582 INFO:teuthology.orchestra.run.smithi090.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-04-24T14:20:10.746 INFO:teuthology.orchestra.run.smithi090.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-04-24T14:20:10.909 INFO:teuthology.orchestra.run.smithi090.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-04-24T14:20:10.960 INFO:teuthology.orchestra.run.smithi090.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-04-24T14:20:13.429 INFO:teuthology.orchestra.run.smithi103.stdout:Fetched 20.0 MB in 3s (6,603 kB/s) 2024-04-24T14:20:13.508 INFO:teuthology.orchestra.run.smithi173.stdout:Fetched 20.0 MB in 3s (6,406 kB/s) 2024-04-24T14:20:13.511 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 20.0 MB in 3s (6,880 kB/s) 2024-04-24T14:20:14.926 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2024-04-24T14:20:14.953 DEBUG:teuthology.orchestra.run.smithi103:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T14:20:14.958 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2024-04-24T14:20:14.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T14:20:15.033 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2024-04-24T14:20:15.070 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2024-04-24T14:20:15.146 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2024-04-24T14:20:15.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T14:20:15.245 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2024-04-24T14:20:15.246 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2024-04-24T14:20:15.252 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2024-04-24T14:20:15.281 INFO:teuthology.orchestra.run.smithi173.stdout:Building dependency tree... 2024-04-24T14:20:15.282 INFO:teuthology.orchestra.run.smithi173.stdout:Reading state information... 2024-04-24T14:20:15.441 INFO:teuthology.orchestra.run.smithi103.stdout:The following additional packages will be installed: 2024-04-24T14:20:15.442 INFO:teuthology.orchestra.run.smithi103.stdout: linux-generic linux-headers-5.4.0-177 linux-headers-5.4.0-177-generic 2024-04-24T14:20:15.442 INFO:teuthology.orchestra.run.smithi103.stdout: linux-headers-generic linux-image-5.4.0-177-generic 2024-04-24T14:20:15.442 INFO:teuthology.orchestra.run.smithi103.stdout: linux-modules-5.4.0-177-generic linux-modules-extra-5.4.0-177-generic 2024-04-24T14:20:15.443 INFO:teuthology.orchestra.run.smithi103.stdout:Suggested packages: 2024-04-24T14:20:15.443 INFO:teuthology.orchestra.run.smithi103.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-04-24T14:20:15.443 INFO:teuthology.orchestra.run.smithi103.stdout:Recommended packages: 2024-04-24T14:20:15.443 INFO:teuthology.orchestra.run.smithi103.stdout: thermald 2024-04-24T14:20:15.473 INFO:teuthology.orchestra.run.smithi090.stdout:Building dependency tree... 2024-04-24T14:20:15.474 INFO:teuthology.orchestra.run.smithi090.stdout:Reading state information... 2024-04-24T14:20:15.484 INFO:teuthology.orchestra.run.smithi173.stdout:The following additional packages will be installed: 2024-04-24T14:20:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: linux-generic linux-headers-5.4.0-177 linux-headers-5.4.0-177-generic 2024-04-24T14:20:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: linux-headers-generic linux-image-5.4.0-177-generic 2024-04-24T14:20:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: linux-modules-5.4.0-177-generic linux-modules-extra-5.4.0-177-generic 2024-04-24T14:20:15.486 INFO:teuthology.orchestra.run.smithi173.stdout:Suggested packages: 2024-04-24T14:20:15.486 INFO:teuthology.orchestra.run.smithi173.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-04-24T14:20:15.487 INFO:teuthology.orchestra.run.smithi173.stdout:Recommended packages: 2024-04-24T14:20:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: thermald 2024-04-24T14:20:15.494 INFO:teuthology.orchestra.run.smithi103.stdout:The following NEW packages will be installed: 2024-04-24T14:20:15.494 INFO:teuthology.orchestra.run.smithi103.stdout: linux-headers-5.4.0-177 linux-headers-5.4.0-177-generic 2024-04-24T14:20:15.494 INFO:teuthology.orchestra.run.smithi103.stdout: linux-image-5.4.0-177-generic linux-modules-5.4.0-177-generic 2024-04-24T14:20:15.494 INFO:teuthology.orchestra.run.smithi103.stdout: linux-modules-extra-5.4.0-177-generic 2024-04-24T14:20:15.495 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages will be upgraded: 2024-04-24T14:20:15.496 INFO:teuthology.orchestra.run.smithi103.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-24T14:20:15.548 INFO:teuthology.orchestra.run.smithi173.stdout:The following NEW packages will be installed: 2024-04-24T14:20:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: linux-headers-5.4.0-177 linux-headers-5.4.0-177-generic 2024-04-24T14:20:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: linux-image-5.4.0-177-generic linux-modules-5.4.0-177-generic 2024-04-24T14:20:15.549 INFO:teuthology.orchestra.run.smithi173.stdout: linux-modules-extra-5.4.0-177-generic 2024-04-24T14:20:15.550 INFO:teuthology.orchestra.run.smithi173.stdout:The following packages will be upgraded: 2024-04-24T14:20:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-24T14:20:15.616 INFO:teuthology.orchestra.run.smithi173.stdout:3 upgraded, 5 newly installed, 0 to remove and 298 not upgraded. 2024-04-24T14:20:15.616 INFO:teuthology.orchestra.run.smithi173.stdout:Need to get 77.1 MB of archives. 2024-04-24T14:20:15.616 INFO:teuthology.orchestra.run.smithi173.stdout:After this operation, 380 MB of additional disk space will be used. 2024-04-24T14:20:15.616 INFO:teuthology.orchestra.run.smithi173.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-177-generic amd64 5.4.0-177.197 [15.0 MB] 2024-04-24T14:20:15.676 INFO:teuthology.orchestra.run.smithi103.stdout:3 upgraded, 5 newly installed, 0 to remove and 298 not upgraded. 2024-04-24T14:20:15.676 INFO:teuthology.orchestra.run.smithi103.stdout:Need to get 77.1 MB of archives. 2024-04-24T14:20:15.676 INFO:teuthology.orchestra.run.smithi103.stdout:After this operation, 380 MB of additional disk space will be used. 2024-04-24T14:20:15.677 INFO:teuthology.orchestra.run.smithi103.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-177-generic amd64 5.4.0-177.197 [15.0 MB] 2024-04-24T14:20:15.677 INFO:teuthology.orchestra.run.smithi090.stdout:The following additional packages will be installed: 2024-04-24T14:20:15.677 INFO:teuthology.orchestra.run.smithi090.stdout: linux-generic linux-headers-5.4.0-177 linux-headers-5.4.0-177-generic 2024-04-24T14:20:15.677 INFO:teuthology.orchestra.run.smithi090.stdout: linux-headers-generic linux-image-5.4.0-177-generic 2024-04-24T14:20:15.677 INFO:teuthology.orchestra.run.smithi090.stdout: linux-modules-5.4.0-177-generic linux-modules-extra-5.4.0-177-generic 2024-04-24T14:20:15.678 INFO:teuthology.orchestra.run.smithi090.stdout:Suggested packages: 2024-04-24T14:20:15.678 INFO:teuthology.orchestra.run.smithi090.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-04-24T14:20:15.679 INFO:teuthology.orchestra.run.smithi090.stdout:Recommended packages: 2024-04-24T14:20:15.679 INFO:teuthology.orchestra.run.smithi090.stdout: thermald 2024-04-24T14:20:15.734 INFO:teuthology.orchestra.run.smithi090.stdout:The following NEW packages will be installed: 2024-04-24T14:20:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: linux-headers-5.4.0-177 linux-headers-5.4.0-177-generic 2024-04-24T14:20:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: linux-image-5.4.0-177-generic linux-modules-5.4.0-177-generic 2024-04-24T14:20:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: linux-modules-extra-5.4.0-177-generic 2024-04-24T14:20:15.736 INFO:teuthology.orchestra.run.smithi090.stdout:The following packages will be upgraded: 2024-04-24T14:20:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-24T14:20:15.833 INFO:teuthology.orchestra.run.smithi090.stdout:3 upgraded, 5 newly installed, 0 to remove and 298 not upgraded. 2024-04-24T14:20:15.833 INFO:teuthology.orchestra.run.smithi090.stdout:Need to get 77.1 MB of archives. 2024-04-24T14:20:15.833 INFO:teuthology.orchestra.run.smithi090.stdout:After this operation, 380 MB of additional disk space will be used. 2024-04-24T14:20:15.834 INFO:teuthology.orchestra.run.smithi090.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-177-generic amd64 5.4.0-177.197 [15.0 MB] 2024-04-24T14:20:16.056 INFO:teuthology.orchestra.run.smithi173.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-177-generic amd64 5.4.0-177.197 [10.5 MB] 2024-04-24T14:20:16.354 INFO:teuthology.orchestra.run.smithi173.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-177-generic amd64 5.4.0-177.197 [39.2 MB] 2024-04-24T14:20:16.824 INFO:teuthology.orchestra.run.smithi103.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-177-generic amd64 5.4.0-177.197 [10.5 MB] 2024-04-24T14:20:17.091 INFO:teuthology.orchestra.run.smithi103.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-177-generic amd64 5.4.0-177.197 [39.2 MB] 2024-04-24T14:20:17.329 INFO:teuthology.orchestra.run.smithi173.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.177.175 [1,900 B] 2024-04-24T14:20:17.329 INFO:teuthology.orchestra.run.smithi173.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.177.175 [2,612 B] 2024-04-24T14:20:17.330 INFO:teuthology.orchestra.run.smithi173.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-177 all 5.4.0-177.197 [11.0 MB] 2024-04-24T14:20:17.454 INFO:teuthology.orchestra.run.smithi090.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-177-generic amd64 5.4.0-177.197 [10.5 MB] 2024-04-24T14:20:17.459 INFO:teuthology.orchestra.run.smithi173.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-177-generic amd64 5.4.0-177.197 [1,377 kB] 2024-04-24T14:20:17.483 INFO:teuthology.orchestra.run.smithi173.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.177.175 [2,480 B] 2024-04-24T14:20:17.809 INFO:teuthology.orchestra.run.smithi090.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-177-generic amd64 5.4.0-177.197 [39.2 MB] 2024-04-24T14:20:18.126 INFO:teuthology.orchestra.run.smithi173.stdout:Fetched 77.1 MB in 2s (40.0 MB/s) 2024-04-24T14:20:18.291 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package linux-modules-5.4.0-177-generic. 2024-04-24T14:20:18.372 INFO:teuthology.orchestra.run.smithi103.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.177.175 [1,900 B] 2024-04-24T14:20:18.372 INFO:teuthology.orchestra.run.smithi103.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.177.175 [2,612 B] 2024-04-24T14:20:18.372 INFO:teuthology.orchestra.run.smithi103.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-177 all 5.4.0-177.197 [11.0 MB] 2024-04-24T14:20:18.666 INFO:teuthology.orchestra.run.smithi090.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.177.175 [1,900 B] 2024-04-24T14:20:18.666 INFO:teuthology.orchestra.run.smithi090.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.177.175 [2,612 B] 2024-04-24T14:20:18.666 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-177 all 5.4.0-177.197 [11.0 MB] 2024-04-24T14:20:18.678 INFO:teuthology.orchestra.run.smithi103.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-177-generic amd64 5.4.0-177.197 [1,377 kB] 2024-04-24T14:20:18.713 INFO:teuthology.orchestra.run.smithi103.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.177.175 [2,480 B] 2024-04-24T14:20:18.790 INFO:teuthology.orchestra.run.smithi090.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-177-generic amd64 5.4.0-177.197 [1,377 kB] 2024-04-24T14:20:18.803 INFO:teuthology.orchestra.run.smithi090.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.177.175 [2,480 B] 2024-04-24T14:20:19.414 INFO:teuthology.orchestra.run.smithi103.stdout:Fetched 77.1 MB in 3s (24.0 MB/s) 2024-04-24T14:20:19.516 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 77.1 MB in 3s (25.2 MB/s) 2024-04-24T14:20:19.584 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package linux-modules-5.4.0-177-generic. 2024-04-24T14:20:19.715 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-modules-5.4.0-177-generic. 2024-04-24T14:20:20.066 INFO:teuthology.orchestra.run.smithi173.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-04-24T14:20:20.071 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../0-linux-modules-5.4.0-177-generic_5.4.0-177.197_amd64.deb ... 2024-04-24T14:20:20.146 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-modules-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:21.307 INFO:teuthology.orchestra.run.smithi103.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-04-24T14:20:21.312 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../0-linux-modules-5.4.0-177-generic_5.4.0-177.197_amd64.deb ... 2024-04-24T14:20:21.390 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-modules-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:22.018 INFO:teuthology.orchestra.run.smithi090.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-04-24T14:20:22.023 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../0-linux-modules-5.4.0-177-generic_5.4.0-177.197_amd64.deb ... 2024-04-24T14:20:22.040 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package linux-image-5.4.0-177-generic. 2024-04-24T14:20:22.053 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../1-linux-image-5.4.0-177-generic_5.4.0-177.197_amd64.deb ... 2024-04-24T14:20:22.073 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-modules-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:22.140 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-image-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:23.202 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-177-generic. 2024-04-24T14:20:23.217 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-177-generic_5.4.0-177.197_amd64.deb ... 2024-04-24T14:20:23.260 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-modules-extra-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:23.417 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package linux-image-5.4.0-177-generic. 2024-04-24T14:20:23.434 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../1-linux-image-5.4.0-177-generic_5.4.0-177.197_amd64.deb ... 2024-04-24T14:20:23.545 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-image-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:24.097 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-image-5.4.0-177-generic. 2024-04-24T14:20:24.113 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../1-linux-image-5.4.0-177-generic_5.4.0-177.197_amd64.deb ... 2024-04-24T14:20:24.215 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-image-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:24.887 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-177-generic. 2024-04-24T14:20:24.901 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-177-generic_5.4.0-177.197_amd64.deb ... 2024-04-24T14:20:24.946 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-modules-extra-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:25.392 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-177-generic. 2024-04-24T14:20:25.406 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-177-generic_5.4.0-177.197_amd64.deb ... 2024-04-24T14:20:25.450 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-modules-extra-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:28.203 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../3-linux-generic_5.4.0.177.175_amd64.deb ... 2024-04-24T14:20:28.321 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-generic (5.4.0.177.175) over (5.4.0.124.125) ... 2024-04-24T14:20:28.654 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.177.175_amd64.deb ... 2024-04-24T14:20:28.791 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-image-generic (5.4.0.177.175) over (5.4.0.124.125) ... 2024-04-24T14:20:29.111 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package linux-headers-5.4.0-177. 2024-04-24T14:20:29.124 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../5-linux-headers-5.4.0-177_5.4.0-177.197_all.deb ... 2024-04-24T14:20:29.161 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-headers-5.4.0-177 (5.4.0-177.197) ... 2024-04-24T14:20:29.720 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../3-linux-generic_5.4.0.177.175_amd64.deb ... 2024-04-24T14:20:29.848 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-generic (5.4.0.177.175) over (5.4.0.124.125) ... 2024-04-24T14:20:30.207 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.177.175_amd64.deb ... 2024-04-24T14:20:30.335 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-image-generic (5.4.0.177.175) over (5.4.0.124.125) ... 2024-04-24T14:20:30.663 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package linux-headers-5.4.0-177. 2024-04-24T14:20:30.678 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../5-linux-headers-5.4.0-177_5.4.0-177.197_all.deb ... 2024-04-24T14:20:30.713 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-headers-5.4.0-177 (5.4.0-177.197) ... 2024-04-24T14:20:31.585 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../3-linux-generic_5.4.0.177.175_amd64.deb ... 2024-04-24T14:20:31.708 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-generic (5.4.0.177.175) over (5.4.0.124.125) ... 2024-04-24T14:20:32.043 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.177.175_amd64.deb ... 2024-04-24T14:20:32.188 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-image-generic (5.4.0.177.175) over (5.4.0.124.125) ... 2024-04-24T14:20:32.534 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-headers-5.4.0-177. 2024-04-24T14:20:32.551 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../5-linux-headers-5.4.0-177_5.4.0-177.197_all.deb ... 2024-04-24T14:20:32.592 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-headers-5.4.0-177 (5.4.0-177.197) ... 2024-04-24T14:20:34.381 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package linux-headers-5.4.0-177-generic. 2024-04-24T14:20:34.403 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../6-linux-headers-5.4.0-177-generic_5.4.0-177.197_amd64.deb ... 2024-04-24T14:20:34.448 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-headers-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:36.114 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package linux-headers-5.4.0-177-generic. 2024-04-24T14:20:36.131 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../6-linux-headers-5.4.0-177-generic_5.4.0-177.197_amd64.deb ... 2024-04-24T14:20:36.169 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-headers-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:37.230 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.177.175_amd64.deb ... 2024-04-24T14:20:37.358 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-headers-generic (5.4.0.177.175) over (5.4.0.124.125) ... 2024-04-24T14:20:37.693 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-modules-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:38.837 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.177.175_amd64.deb ... 2024-04-24T14:20:38.934 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-headers-5.4.0-177 (5.4.0-177.197) ... 2024-04-24T14:20:38.972 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-headers-generic (5.4.0.177.175) over (5.4.0.124.125) ... 2024-04-24T14:20:39.076 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-headers-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:39.120 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-headers-5.4.0-177-generic. 2024-04-24T14:20:39.140 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../6-linux-headers-5.4.0-177-generic_5.4.0-177.197_amd64.deb ... 2024-04-24T14:20:39.187 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-headers-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:39.203 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-image-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:39.295 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-modules-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:40.345 INFO:teuthology.orchestra.run.smithi173.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-177-generic 2024-04-24T14:20:40.345 INFO:teuthology.orchestra.run.smithi173.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-177-generic 2024-04-24T14:20:40.471 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-headers-generic (5.4.0.177.175) ... 2024-04-24T14:20:40.572 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-modules-extra-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:40.778 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-headers-5.4.0-177 (5.4.0-177.197) ... 2024-04-24T14:20:40.904 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-headers-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:41.030 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-image-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:41.535 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.177.175_amd64.deb ... 2024-04-24T14:20:41.663 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-headers-generic (5.4.0.177.175) over (5.4.0.124.125) ... 2024-04-24T14:20:41.732 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-image-generic (5.4.0.177.175) ... 2024-04-24T14:20:41.858 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-generic (5.4.0.177.175) ... 2024-04-24T14:20:41.985 INFO:teuthology.orchestra.run.smithi173.stdout:Processing triggers for linux-image-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:42.024 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-modules-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:42.057 INFO:teuthology.orchestra.run.smithi173.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-24T14:20:42.058 INFO:teuthology.orchestra.run.smithi173.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-177-generic 2024-04-24T14:20:42.172 INFO:teuthology.orchestra.run.smithi103.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-177-generic 2024-04-24T14:20:42.172 INFO:teuthology.orchestra.run.smithi103.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-177-generic 2024-04-24T14:20:42.316 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-headers-generic (5.4.0.177.175) ... 2024-04-24T14:20:42.442 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-modules-extra-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:43.610 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-image-generic (5.4.0.177.175) ... 2024-04-24T14:20:43.736 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-generic (5.4.0.177.175) ... 2024-04-24T14:20:43.862 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for linux-image-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:43.932 INFO:teuthology.orchestra.run.smithi103.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-24T14:20:43.932 INFO:teuthology.orchestra.run.smithi103.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-177-generic 2024-04-24T14:20:44.486 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-headers-5.4.0-177 (5.4.0-177.197) ... 2024-04-24T14:20:44.607 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-headers-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:44.734 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-image-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:45.901 INFO:teuthology.orchestra.run.smithi090.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-177-generic 2024-04-24T14:20:45.902 INFO:teuthology.orchestra.run.smithi090.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-177-generic 2024-04-24T14:20:46.070 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-headers-generic (5.4.0.177.175) ... 2024-04-24T14:20:46.197 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-modules-extra-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:47.390 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-image-generic (5.4.0.177.175) ... 2024-04-24T14:20:47.516 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-generic (5.4.0.177.175) ... 2024-04-24T14:20:47.635 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for linux-image-5.4.0-177-generic (5.4.0-177.197) ... 2024-04-24T14:20:47.712 INFO:teuthology.orchestra.run.smithi090.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-24T14:20:47.712 INFO:teuthology.orchestra.run.smithi090.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-177-generic 2024-04-24T14:21:04.558 INFO:teuthology.orchestra.run.smithi173.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-24T14:21:04.559 INFO:teuthology.orchestra.run.smithi173.stdout:Sourcing file `/etc/default/grub' 2024-04-24T14:21:04.570 INFO:teuthology.orchestra.run.smithi173.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T14:21:04.594 INFO:teuthology.orchestra.run.smithi173.stdout:Generating grub configuration file ... 2024-04-24T14:21:04.875 INFO:teuthology.orchestra.run.smithi173.stdout:Found linux image: /boot/vmlinuz-5.4.0-177-generic 2024-04-24T14:21:04.899 INFO:teuthology.orchestra.run.smithi173.stdout:Found initrd image: /boot/initrd.img-5.4.0-177-generic 2024-04-24T14:21:05.269 INFO:teuthology.orchestra.run.smithi173.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-04-24T14:21:05.280 INFO:teuthology.orchestra.run.smithi173.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-04-24T14:21:05.816 INFO:teuthology.orchestra.run.smithi103.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-24T14:21:05.816 INFO:teuthology.orchestra.run.smithi103.stdout:Sourcing file `/etc/default/grub' 2024-04-24T14:21:05.824 INFO:teuthology.orchestra.run.smithi103.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T14:21:05.832 INFO:teuthology.orchestra.run.smithi103.stdout:Generating grub configuration file ... 2024-04-24T14:21:06.152 INFO:teuthology.orchestra.run.smithi103.stdout:Found linux image: /boot/vmlinuz-5.4.0-177-generic 2024-04-24T14:21:06.178 INFO:teuthology.orchestra.run.smithi103.stdout:Found initrd image: /boot/initrd.img-5.4.0-177-generic 2024-04-24T14:21:06.556 INFO:teuthology.orchestra.run.smithi103.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-04-24T14:21:06.569 INFO:teuthology.orchestra.run.smithi103.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-04-24T14:21:07.791 INFO:teuthology.orchestra.run.smithi173.stdout:done 2024-04-24T14:21:08.229 DEBUG:teuthology.orchestra.run.smithi173:> dpkg -s linux-image-generic 2024-04-24T14:21:08.281 INFO:teuthology.orchestra.run.smithi173.stdout:Package: linux-image-generic 2024-04-24T14:21:08.281 INFO:teuthology.orchestra.run.smithi173.stdout:Status: install ok installed 2024-04-24T14:21:08.281 INFO:teuthology.orchestra.run.smithi173.stdout:Priority: optional 2024-04-24T14:21:08.281 INFO:teuthology.orchestra.run.smithi173.stdout:Section: kernel 2024-04-24T14:21:08.281 INFO:teuthology.orchestra.run.smithi173.stdout:Installed-Size: 20 2024-04-24T14:21:08.281 INFO:teuthology.orchestra.run.smithi173.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T14:21:08.282 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: amd64 2024-04-24T14:21:08.282 INFO:teuthology.orchestra.run.smithi173.stdout:Source: linux-meta 2024-04-24T14:21:08.282 INFO:teuthology.orchestra.run.smithi173.stdout:Version: 5.4.0.177.175 2024-04-24T14:21:08.282 INFO:teuthology.orchestra.run.smithi173.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-04-24T14:21:08.282 INFO:teuthology.orchestra.run.smithi173.stdout:Depends: linux-image-5.4.0-177-generic, linux-modules-extra-5.4.0-177-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T14:21:08.282 INFO:teuthology.orchestra.run.smithi173.stdout:Recommends: thermald 2024-04-24T14:21:08.282 INFO:teuthology.orchestra.run.smithi173.stdout:Description: Generic Linux kernel image 2024-04-24T14:21:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: This package will always depend on the latest generic kernel image 2024-04-24T14:21:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: available. 2024-04-24T14:21:08.282 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-177-generic 2024-04-24T14:21:08.282 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-24T14:21:08.283 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-24T14:21:08.283 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-04-24T14:21:08.283 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2024-04-24T14:21:08.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-get clean 2024-04-24T14:21:08.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-get update 2024-04-24T14:21:08.518 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-04-24T14:21:08.526 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-04-24T14:21:08.546 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-04-24T14:21:08.598 INFO:teuthology.orchestra.run.smithi103.stdout:done 2024-04-24T14:21:08.608 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-04-24T14:21:09.043 DEBUG:teuthology.orchestra.run.smithi103:> dpkg -s linux-image-generic 2024-04-24T14:21:09.091 INFO:teuthology.orchestra.run.smithi103.stdout:Package: linux-image-generic 2024-04-24T14:21:09.092 INFO:teuthology.orchestra.run.smithi103.stdout:Status: install ok installed 2024-04-24T14:21:09.092 INFO:teuthology.orchestra.run.smithi103.stdout:Priority: optional 2024-04-24T14:21:09.092 INFO:teuthology.orchestra.run.smithi103.stdout:Section: kernel 2024-04-24T14:21:09.092 INFO:teuthology.orchestra.run.smithi103.stdout:Installed-Size: 20 2024-04-24T14:21:09.092 INFO:teuthology.orchestra.run.smithi103.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T14:21:09.092 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: amd64 2024-04-24T14:21:09.092 INFO:teuthology.orchestra.run.smithi103.stdout:Source: linux-meta 2024-04-24T14:21:09.092 INFO:teuthology.orchestra.run.smithi103.stdout:Version: 5.4.0.177.175 2024-04-24T14:21:09.092 INFO:teuthology.orchestra.run.smithi103.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-04-24T14:21:09.092 INFO:teuthology.orchestra.run.smithi103.stdout:Depends: linux-image-5.4.0-177-generic, linux-modules-extra-5.4.0-177-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T14:21:09.092 INFO:teuthology.orchestra.run.smithi103.stdout:Recommends: thermald 2024-04-24T14:21:09.092 INFO:teuthology.orchestra.run.smithi103.stdout:Description: Generic Linux kernel image 2024-04-24T14:21:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: This package will always depend on the latest generic kernel image 2024-04-24T14:21:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: available. 2024-04-24T14:21:09.093 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-177-generic 2024-04-24T14:21:09.093 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-24T14:21:09.093 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-24T14:21:09.093 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-24T14:21:09.093 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=None, version=distro) 2024-04-24T14:21:09.093 DEBUG:teuthology.orchestra.run.smithi103:> sudo apt-get clean 2024-04-24T14:21:09.171 DEBUG:teuthology.orchestra.run.smithi103:> sudo apt-get update 2024-04-24T14:21:09.359 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-04-24T14:21:09.382 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-04-24T14:21:09.417 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-04-24T14:21:09.417 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-04-24T14:21:09.844 INFO:teuthology.orchestra.run.smithi090.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-24T14:21:09.844 INFO:teuthology.orchestra.run.smithi090.stdout:Sourcing file `/etc/default/grub' 2024-04-24T14:21:09.863 INFO:teuthology.orchestra.run.smithi090.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T14:21:09.882 INFO:teuthology.orchestra.run.smithi090.stdout:Generating grub configuration file ... 2024-04-24T14:21:10.307 INFO:teuthology.orchestra.run.smithi090.stdout:Found linux image: /boot/vmlinuz-5.4.0-177-generic 2024-04-24T14:21:10.332 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2024-04-24T14:21:10.348 INFO:teuthology.orchestra.run.smithi090.stdout:Found initrd image: /boot/initrd.img-5.4.0-177-generic 2024-04-24T14:21:10.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T14:21:10.426 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2024-04-24T14:21:10.613 INFO:teuthology.orchestra.run.smithi173.stdout:Building dependency tree... 2024-04-24T14:21:10.614 INFO:teuthology.orchestra.run.smithi173.stdout:Reading state information... 2024-04-24T14:21:10.834 INFO:teuthology.orchestra.run.smithi173.stdout:linux-image-generic is already the newest version (5.4.0.177.175). 2024-04-24T14:21:10.834 INFO:teuthology.orchestra.run.smithi173.stdout:0 upgraded, 0 newly installed, 0 to remove and 298 not upgraded. 2024-04-24T14:21:10.836 DEBUG:teuthology.orchestra.run.smithi173:> dpkg -s linux-image-generic 2024-04-24T14:21:10.854 INFO:teuthology.orchestra.run.smithi173.stdout:Package: linux-image-generic 2024-04-24T14:21:10.854 INFO:teuthology.orchestra.run.smithi173.stdout:Status: install ok installed 2024-04-24T14:21:10.854 INFO:teuthology.orchestra.run.smithi173.stdout:Priority: optional 2024-04-24T14:21:10.854 INFO:teuthology.orchestra.run.smithi173.stdout:Section: kernel 2024-04-24T14:21:10.854 INFO:teuthology.orchestra.run.smithi173.stdout:Installed-Size: 20 2024-04-24T14:21:10.854 INFO:teuthology.orchestra.run.smithi173.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T14:21:10.855 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: amd64 2024-04-24T14:21:10.855 INFO:teuthology.orchestra.run.smithi173.stdout:Source: linux-meta 2024-04-24T14:21:10.855 INFO:teuthology.orchestra.run.smithi173.stdout:Version: 5.4.0.177.175 2024-04-24T14:21:10.855 INFO:teuthology.orchestra.run.smithi173.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-04-24T14:21:10.855 INFO:teuthology.orchestra.run.smithi173.stdout:Depends: linux-image-5.4.0-177-generic, linux-modules-extra-5.4.0-177-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T14:21:10.855 INFO:teuthology.orchestra.run.smithi173.stdout:Recommends: thermald 2024-04-24T14:21:10.855 INFO:teuthology.orchestra.run.smithi173.stdout:Description: Generic Linux kernel image 2024-04-24T14:21:10.855 INFO:teuthology.orchestra.run.smithi173.stdout: This package will always depend on the latest generic kernel image 2024-04-24T14:21:10.855 INFO:teuthology.orchestra.run.smithi173.stdout: available. 2024-04-24T14:21:10.855 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-04-24T14:21:10.862 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.mKv6pWQpJE 2024-04-24T14:21:10.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub/grub.cfg /tmp/tmp.mKv6pWQpJE 2024-04-24T14:21:10.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.mKv6pWQpJE 2024-04-24T14:21:10.925 INFO:teuthology.orchestra.run.smithi090.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-04-24T14:21:10.935 INFO:teuthology.orchestra.run.smithi090.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-04-24T14:21:10.959 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.mKv6pWQpJE is 9KB 2024-04-24T14:21:10.974 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.mKv6pWQpJE 2024-04-24T14:21:10.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-24T14:21:11.039 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-24T14:21:11.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-24T14:21:11.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-24T14:21:11.071 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-177-generic 2024-04-24T14:21:11.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo update-grub 2024-04-24T14:21:11.181 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2024-04-24T14:21:11.206 DEBUG:teuthology.orchestra.run.smithi103:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T14:21:11.275 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2024-04-24T14:21:11.462 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2024-04-24T14:21:11.463 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2024-04-24T14:21:11.684 INFO:teuthology.orchestra.run.smithi103.stdout:linux-image-generic is already the newest version (5.4.0.177.175). 2024-04-24T14:21:11.684 INFO:teuthology.orchestra.run.smithi103.stdout:0 upgraded, 0 newly installed, 0 to remove and 298 not upgraded. 2024-04-24T14:21:11.686 DEBUG:teuthology.orchestra.run.smithi103:> dpkg -s linux-image-generic 2024-04-24T14:21:11.706 INFO:teuthology.orchestra.run.smithi103.stdout:Package: linux-image-generic 2024-04-24T14:21:11.707 INFO:teuthology.orchestra.run.smithi103.stdout:Status: install ok installed 2024-04-24T14:21:11.707 INFO:teuthology.orchestra.run.smithi103.stdout:Priority: optional 2024-04-24T14:21:11.707 INFO:teuthology.orchestra.run.smithi103.stdout:Section: kernel 2024-04-24T14:21:11.707 INFO:teuthology.orchestra.run.smithi103.stdout:Installed-Size: 20 2024-04-24T14:21:11.707 INFO:teuthology.orchestra.run.smithi103.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T14:21:11.707 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: amd64 2024-04-24T14:21:11.707 INFO:teuthology.orchestra.run.smithi103.stdout:Source: linux-meta 2024-04-24T14:21:11.707 INFO:teuthology.orchestra.run.smithi103.stdout:Version: 5.4.0.177.175 2024-04-24T14:21:11.707 INFO:teuthology.orchestra.run.smithi103.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-04-24T14:21:11.707 INFO:teuthology.orchestra.run.smithi103.stdout:Depends: linux-image-5.4.0-177-generic, linux-modules-extra-5.4.0-177-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T14:21:11.707 INFO:teuthology.orchestra.run.smithi103.stdout:Recommends: thermald 2024-04-24T14:21:11.707 INFO:teuthology.orchestra.run.smithi103.stdout:Description: Generic Linux kernel image 2024-04-24T14:21:11.707 INFO:teuthology.orchestra.run.smithi103.stdout: This package will always depend on the latest generic kernel image 2024-04-24T14:21:11.708 INFO:teuthology.orchestra.run.smithi103.stdout: available. 2024-04-24T14:21:11.708 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-04-24T14:21:11.713 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.kXlClPrWUZ 2024-04-24T14:21:11.714 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub/grub.cfg /tmp/tmp.kXlClPrWUZ 2024-04-24T14:21:11.727 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.kXlClPrWUZ 2024-04-24T14:21:11.805 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.kXlClPrWUZ is 9KB 2024-04-24T14:21:11.855 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.kXlClPrWUZ 2024-04-24T14:21:11.864 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-24T14:21:11.880 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:21:11.880 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-24T14:21:11.880 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-24T14:21:11.961 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-177-generic 2024-04-24T14:21:11.961 DEBUG:teuthology.orchestra.run.smithi103:> sudo update-grub 2024-04-24T14:21:12.442 INFO:teuthology.orchestra.run.smithi173.stderr:Sourcing file `/etc/default/grub' 2024-04-24T14:21:12.452 INFO:teuthology.orchestra.run.smithi173.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T14:21:12.467 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-04-24T14:21:12.713 INFO:teuthology.orchestra.run.smithi173.stderr:Found linux image: /boot/vmlinuz-5.4.0-177-generic 2024-04-24T14:21:12.724 INFO:teuthology.orchestra.run.smithi173.stderr:Found initrd image: /boot/initrd.img-5.4.0-177-generic 2024-04-24T14:21:13.096 INFO:teuthology.orchestra.run.smithi173.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-04-24T14:21:13.107 INFO:teuthology.orchestra.run.smithi173.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-04-24T14:21:13.372 INFO:teuthology.orchestra.run.smithi103.stderr:Sourcing file `/etc/default/grub' 2024-04-24T14:21:13.381 INFO:teuthology.orchestra.run.smithi103.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T14:21:13.391 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-04-24T14:21:13.656 INFO:teuthology.orchestra.run.smithi103.stderr:Found linux image: /boot/vmlinuz-5.4.0-177-generic 2024-04-24T14:21:13.667 INFO:teuthology.orchestra.run.smithi103.stderr:Found initrd image: /boot/initrd.img-5.4.0-177-generic 2024-04-24T14:21:13.992 INFO:teuthology.orchestra.run.smithi090.stdout:done 2024-04-24T14:21:14.027 INFO:teuthology.orchestra.run.smithi103.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-04-24T14:21:14.038 INFO:teuthology.orchestra.run.smithi103.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-04-24T14:21:14.462 DEBUG:teuthology.orchestra.run.smithi090:> dpkg -s linux-image-generic 2024-04-24T14:21:14.516 INFO:teuthology.orchestra.run.smithi090.stdout:Package: linux-image-generic 2024-04-24T14:21:14.516 INFO:teuthology.orchestra.run.smithi090.stdout:Status: install ok installed 2024-04-24T14:21:14.516 INFO:teuthology.orchestra.run.smithi090.stdout:Priority: optional 2024-04-24T14:21:14.516 INFO:teuthology.orchestra.run.smithi090.stdout:Section: kernel 2024-04-24T14:21:14.516 INFO:teuthology.orchestra.run.smithi090.stdout:Installed-Size: 20 2024-04-24T14:21:14.516 INFO:teuthology.orchestra.run.smithi090.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T14:21:14.517 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: amd64 2024-04-24T14:21:14.517 INFO:teuthology.orchestra.run.smithi090.stdout:Source: linux-meta 2024-04-24T14:21:14.517 INFO:teuthology.orchestra.run.smithi090.stdout:Version: 5.4.0.177.175 2024-04-24T14:21:14.517 INFO:teuthology.orchestra.run.smithi090.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-04-24T14:21:14.517 INFO:teuthology.orchestra.run.smithi090.stdout:Depends: linux-image-5.4.0-177-generic, linux-modules-extra-5.4.0-177-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T14:21:14.517 INFO:teuthology.orchestra.run.smithi090.stdout:Recommends: thermald 2024-04-24T14:21:14.517 INFO:teuthology.orchestra.run.smithi090.stdout:Description: Generic Linux kernel image 2024-04-24T14:21:14.517 INFO:teuthology.orchestra.run.smithi090.stdout: This package will always depend on the latest generic kernel image 2024-04-24T14:21:14.517 INFO:teuthology.orchestra.run.smithi090.stdout: available. 2024-04-24T14:21:14.517 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-177-generic 2024-04-24T14:21:14.517 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-24T14:21:14.518 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-24T14:21:14.518 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-24T14:21:14.518 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=None, version=distro) 2024-04-24T14:21:14.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get clean 2024-04-24T14:21:14.570 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-04-24T14:21:14.572 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-04-24T14:21:14.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get update 2024-04-24T14:21:14.895 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-04-24T14:21:14.895 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-04-24T14:21:14.966 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-04-24T14:21:15.042 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-04-24T14:21:15.043 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-04-24T14:21:15.048 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-04-24T14:21:16.740 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2024-04-24T14:21:16.765 DEBUG:teuthology.orchestra.run.smithi090:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T14:21:16.835 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2024-04-24T14:21:17.045 INFO:teuthology.orchestra.run.smithi090.stdout:Building dependency tree... 2024-04-24T14:21:17.045 INFO:teuthology.orchestra.run.smithi090.stdout:Reading state information... 2024-04-24T14:21:17.267 INFO:teuthology.orchestra.run.smithi090.stdout:linux-image-generic is already the newest version (5.4.0.177.175). 2024-04-24T14:21:17.267 INFO:teuthology.orchestra.run.smithi090.stdout:0 upgraded, 0 newly installed, 0 to remove and 298 not upgraded. 2024-04-24T14:21:17.269 DEBUG:teuthology.orchestra.run.smithi090:> dpkg -s linux-image-generic 2024-04-24T14:21:17.292 INFO:teuthology.orchestra.run.smithi090.stdout:Package: linux-image-generic 2024-04-24T14:21:17.292 INFO:teuthology.orchestra.run.smithi090.stdout:Status: install ok installed 2024-04-24T14:21:17.292 INFO:teuthology.orchestra.run.smithi090.stdout:Priority: optional 2024-04-24T14:21:17.292 INFO:teuthology.orchestra.run.smithi090.stdout:Section: kernel 2024-04-24T14:21:17.292 INFO:teuthology.orchestra.run.smithi090.stdout:Installed-Size: 20 2024-04-24T14:21:17.292 INFO:teuthology.orchestra.run.smithi090.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T14:21:17.292 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: amd64 2024-04-24T14:21:17.293 INFO:teuthology.orchestra.run.smithi090.stdout:Source: linux-meta 2024-04-24T14:21:17.293 INFO:teuthology.orchestra.run.smithi090.stdout:Version: 5.4.0.177.175 2024-04-24T14:21:17.293 INFO:teuthology.orchestra.run.smithi090.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-04-24T14:21:17.293 INFO:teuthology.orchestra.run.smithi090.stdout:Depends: linux-image-5.4.0-177-generic, linux-modules-extra-5.4.0-177-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T14:21:17.293 INFO:teuthology.orchestra.run.smithi090.stdout:Recommends: thermald 2024-04-24T14:21:17.293 INFO:teuthology.orchestra.run.smithi090.stdout:Description: Generic Linux kernel image 2024-04-24T14:21:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: This package will always depend on the latest generic kernel image 2024-04-24T14:21:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: available. 2024-04-24T14:21:17.293 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-04-24T14:21:17.339 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.WmBX95R5ZW 2024-04-24T14:21:17.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub/grub.cfg /tmp/tmp.WmBX95R5ZW 2024-04-24T14:21:17.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.WmBX95R5ZW 2024-04-24T14:21:17.396 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.WmBX95R5ZW is 9KB 2024-04-24T14:21:17.410 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.WmBX95R5ZW 2024-04-24T14:21:17.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-24T14:21:17.474 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:21:17.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-24T14:21:17.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-24T14:21:17.502 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-177-generic 2024-04-24T14:21:17.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo update-grub 2024-04-24T14:21:19.088 INFO:teuthology.orchestra.run.smithi090.stderr:Sourcing file `/etc/default/grub' 2024-04-24T14:21:19.092 INFO:teuthology.orchestra.run.smithi090.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T14:21:19.108 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-04-24T14:21:19.540 INFO:teuthology.orchestra.run.smithi090.stderr:Found linux image: /boot/vmlinuz-5.4.0-177-generic 2024-04-24T14:21:19.552 INFO:teuthology.orchestra.run.smithi090.stderr:Found initrd image: /boot/initrd.img-5.4.0-177-generic 2024-04-24T14:21:20.060 INFO:teuthology.orchestra.run.smithi090.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-04-24T14:21:20.071 INFO:teuthology.orchestra.run.smithi090.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-04-24T14:21:21.842 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-04-24T14:21:21.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2024-04-24T14:21:44.594 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-24T14:21:44.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-24T14:21:44.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:21:45.046 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-24T14:21:45.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-24T14:21:45.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:21:51.850 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-24T14:21:51.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-24T14:21:51.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:22:03.039 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-04-24T14:22:12.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-24T14:22:12.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:22:15.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-04-24T14:22:19.678 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-04-24T14:22:26.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-04-24T14:22:27.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-24T14:22:27.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:22:28.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-24T14:22:28.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:22:35.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-24T14:22:35.339 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:22:38.402 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-04-24T14:22:45.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-04-24T14:22:50.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-24T14:22:50.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:23:00.542 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-24T14:23:00.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:23:00.800 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-04-24T14:23:01.359 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-24T14:23:01.359 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.4.0-177-generic"... 2024-04-24T14:23:01.360 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-04-24T14:23:01.366 INFO:teuthology.orchestra.run.smithi173.stdout:5.4.0-177-generic 2024-04-24T14:23:01.367 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-177-generic vs 5.4.0-177-generic 2024-04-24T14:23:01.367 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T14:23:01.367 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-24T14:23:02.367 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2024-04-24T14:23:02.368 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T14:23:02.451 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-04-24T14:23:02.484 DEBUG:teuthology.parallel:result is None 2024-04-24T14:23:22.141 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-04-24T14:23:22.799 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-24T14:23:22.799 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-177-generic"... 2024-04-24T14:23:22.799 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-04-24T14:23:22.806 INFO:teuthology.orchestra.run.smithi090.stdout:5.4.0-177-generic 2024-04-24T14:23:22.806 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-177-generic vs 5.4.0-177-generic 2024-04-24T14:23:22.806 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T14:23:22.806 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-24T14:23:23.807 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-24T14:23:23.808 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T14:23:23.932 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2024-04-24T14:23:23.958 DEBUG:teuthology.parallel:result is None 2024-04-24T14:23:28.683 DEBUG:teuthology.orchestra.remote:timed out 2024-04-24T14:23:40.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-24T14:23:40.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:23:40.932 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-04-24T14:23:41.523 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-24T14:23:41.524 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-177-generic"... 2024-04-24T14:23:41.524 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-04-24T14:23:41.531 INFO:teuthology.orchestra.run.smithi103.stdout:5.4.0-177-generic 2024-04-24T14:23:41.531 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-177-generic vs 5.4.0-177-generic 2024-04-24T14:23:41.531 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T14:23:41.531 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-24T14:23:42.532 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-24T14:23:42.532 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T14:23:42.615 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2024-04-24T14:23:42.640 DEBUG:teuthology.parallel:result is None 2024-04-24T14:23:42.641 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T14:23:42.649 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T14:23:42.649 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T14:23:42.652 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T14:23:42.654 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T14:23:42.663 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T14:23:42.688 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T14:23:42.721 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T14:23:42.721 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T14:23:42.724 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T14:23:42.726 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T14:23:42.745 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T14:23:42.754 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T14:23:42.754 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T14:23:42.756 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T14:23:42.762 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T14:23:42.774 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T14:23:42.776 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T14:23:42.785 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T14:23:42.788 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T14:23:42.806 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T14:23:42.817 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T14:23:42.819 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T14:23:42.827 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T14:23:42.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T14:23:42.834 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T14:23:42.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T14:23:42.851 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T14:23:42.860 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T14:23:42.861 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T14:23:42.864 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T14:23:42.866 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T14:23:42.874 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T14:23:42.914 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T14:23:42.921 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:23:42.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T14:23:42.944 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T14:23:42.952 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T14:23:43.001 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:23:43.001 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T14:23:43.022 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T14:23:43.029 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T14:23:43.037 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-24T14:23:43.037 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T14:23:43.056 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-04-24T14:23:43.058 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-04-24T14:23:43.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-04-24T14:23:43.154 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T14:23:43.162 INFO:teuthology.task.internal:Starting timer... 2024-04-24T14:23:43.162 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T14:23:43.234 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T14:23:43.266 DEBUG:teuthology.task.selinux:Excluding smithi090: OS 'ubuntu' does not support SELinux 2024-04-24T14:23:43.266 DEBUG:teuthology.task.selinux:Excluding smithi103: OS 'ubuntu' does not support SELinux 2024-04-24T14:23:43.266 DEBUG:teuthology.task.selinux:Excluding smithi173: OS 'ubuntu' does not support SELinux 2024-04-24T14:23:43.266 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T14:23:43.266 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-24T14:23:43.267 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T14:23:43.267 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T14:23:43.309 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-24T14:23:43.315 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-24T14:23:43.442 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T14:23:43.462 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T14:23:43.464 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi090.front.sepia.ceph.com,smithi103.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T14:29:18.553 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi090.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-04-24T14:29:18.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-24T14:29:18.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:29:18.656 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-04-24T14:29:18.776 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-04-24T14:29:18.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-24T14:29:18.777 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:29:18.878 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-04-24T14:29:19.004 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-04-24T14:29:19.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-24T14:29:19.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T14:29:19.112 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-04-24T14:29:19.240 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-24T14:29:19.241 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T14:29:19.257 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T14:29:19.257 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T14:29:19.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T14:29:19.260 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T14:29:19.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T14:29:19.263 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T14:29:19.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T14:29:19.292 INFO:teuthology.orchestra.run.smithi090.stdout:24 Apr 14:29:19 ntpd[13027]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-04-24T14:29:19.292 INFO:teuthology.orchestra.run.smithi090.stdout:24 Apr 14:29:19 ntpd[13027]: Command line: ntpd -gq 2024-04-24T14:29:19.293 INFO:teuthology.orchestra.run.smithi090.stdout:24 Apr 14:29:19 ntpd[13027]: proto: precision = 0.084 usec (-23) 2024-04-24T14:29:19.293 INFO:teuthology.orchestra.run.smithi090.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T14:29:19.294 INFO:teuthology.orchestra.run.smithi090.stdout:24 Apr 14:29:19 ntpd[13027]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T14:29:19.294 INFO:teuthology.orchestra.run.smithi090.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-24T14:29:19.296 INFO:teuthology.orchestra.run.smithi090.stdout:24 Apr 14:29:19 ntpd[13027]: restrict ::: KOD does nothing without LIMITED. 2024-04-24T14:29:19.296 INFO:teuthology.orchestra.run.smithi090.stdout:24 Apr 14:29:19 ntpd[13027]: Listen and drop on 0 v6wildcard [::]:123 2024-04-24T14:29:19.296 INFO:teuthology.orchestra.run.smithi090.stdout:24 Apr 14:29:19 ntpd[13027]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-24T14:29:19.296 INFO:teuthology.orchestra.run.smithi090.stdout:24 Apr 14:29:19 ntpd[13027]: Listen normally on 2 lo 127.0.0.1:123 2024-04-24T14:29:19.296 INFO:teuthology.orchestra.run.smithi090.stdout:24 Apr 14:29:19 ntpd[13027]: Listen normally on 3 enp3s0f1 172.21.15.90:123 2024-04-24T14:29:19.296 INFO:teuthology.orchestra.run.smithi090.stdout:24 Apr 14:29:19 ntpd[13027]: Listen normally on 4 lo [::1]:123 2024-04-24T14:29:19.296 INFO:teuthology.orchestra.run.smithi090.stdout:24 Apr 14:29:19 ntpd[13027]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d767%5]:123 2024-04-24T14:29:19.296 INFO:teuthology.orchestra.run.smithi090.stdout:24 Apr 14:29:19 ntpd[13027]: Listening on routing socket on fd #22 for interface updates 2024-04-24T14:29:19.297 INFO:teuthology.orchestra.run.smithi103.stdout:24 Apr 14:29:19 ntpd[12972]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-04-24T14:29:19.297 INFO:teuthology.orchestra.run.smithi103.stdout:24 Apr 14:29:19 ntpd[12972]: Command line: ntpd -gq 2024-04-24T14:29:19.298 INFO:teuthology.orchestra.run.smithi173.stdout:24 Apr 14:29:19 ntpd[12966]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-04-24T14:29:19.298 INFO:teuthology.orchestra.run.smithi173.stdout:24 Apr 14:29:19 ntpd[12966]: Command line: ntpd -gq 2024-04-24T14:29:19.299 INFO:teuthology.orchestra.run.smithi103.stdout:24 Apr 14:29:19 ntpd[12972]: proto: precision = 0.117 usec (-23) 2024-04-24T14:29:19.299 INFO:teuthology.orchestra.run.smithi173.stdout:24 Apr 14:29:19 ntpd[12966]: proto: precision = 0.117 usec (-23) 2024-04-24T14:29:19.299 INFO:teuthology.orchestra.run.smithi103.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T14:29:19.300 INFO:teuthology.orchestra.run.smithi103.stdout:24 Apr 14:29:19 ntpd[12972]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T14:29:19.300 INFO:teuthology.orchestra.run.smithi103.stdout:24 Apr 14:29:19 ntpd[12972]: restrict ::: KOD does nothing without LIMITED. 2024-04-24T14:29:19.300 INFO:teuthology.orchestra.run.smithi103.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-24T14:29:19.301 INFO:teuthology.orchestra.run.smithi103.stdout:24 Apr 14:29:19 ntpd[12972]: Listen and drop on 0 v6wildcard [::]:123 2024-04-24T14:29:19.301 INFO:teuthology.orchestra.run.smithi103.stdout:24 Apr 14:29:19 ntpd[12972]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-24T14:29:19.301 INFO:teuthology.orchestra.run.smithi103.stdout:24 Apr 14:29:19 ntpd[12972]: Listen normally on 2 lo 127.0.0.1:123 2024-04-24T14:29:19.301 INFO:teuthology.orchestra.run.smithi103.stdout:24 Apr 14:29:19 ntpd[12972]: Listen normally on 3 enp3s0f1 172.21.15.103:123 2024-04-24T14:29:19.301 INFO:teuthology.orchestra.run.smithi103.stdout:24 Apr 14:29:19 ntpd[12972]: Listen normally on 4 lo [::1]:123 2024-04-24T14:29:19.301 INFO:teuthology.orchestra.run.smithi103.stdout:24 Apr 14:29:19 ntpd[12972]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cc0b%5]:123 2024-04-24T14:29:19.301 INFO:teuthology.orchestra.run.smithi103.stdout:24 Apr 14:29:19 ntpd[12972]: Listening on routing socket on fd #22 for interface updates 2024-04-24T14:29:19.302 INFO:teuthology.orchestra.run.smithi173.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T14:29:19.302 INFO:teuthology.orchestra.run.smithi173.stdout:24 Apr 14:29:19 ntpd[12966]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T14:29:19.302 INFO:teuthology.orchestra.run.smithi173.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-24T14:29:19.303 INFO:teuthology.orchestra.run.smithi173.stdout:24 Apr 14:29:19 ntpd[12966]: restrict ::: KOD does nothing without LIMITED. 2024-04-24T14:29:19.304 INFO:teuthology.orchestra.run.smithi173.stdout:24 Apr 14:29:19 ntpd[12966]: Listen and drop on 0 v6wildcard [::]:123 2024-04-24T14:29:19.304 INFO:teuthology.orchestra.run.smithi173.stdout:24 Apr 14:29:19 ntpd[12966]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-24T14:29:19.304 INFO:teuthology.orchestra.run.smithi173.stdout:24 Apr 14:29:19 ntpd[12966]: Listen normally on 2 lo 127.0.0.1:123 2024-04-24T14:29:19.304 INFO:teuthology.orchestra.run.smithi173.stdout:24 Apr 14:29:19 ntpd[12966]: Listen normally on 3 enp3s0f1 172.21.15.173:123 2024-04-24T14:29:19.304 INFO:teuthology.orchestra.run.smithi173.stdout:24 Apr 14:29:19 ntpd[12966]: Listen normally on 4 lo [::1]:123 2024-04-24T14:29:19.304 INFO:teuthology.orchestra.run.smithi173.stdout:24 Apr 14:29:19 ntpd[12966]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7259%5]:123 2024-04-24T14:29:19.304 INFO:teuthology.orchestra.run.smithi173.stdout:24 Apr 14:29:19 ntpd[12966]: Listening on routing socket on fd #22 for interface updates 2024-04-24T14:29:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:20 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:20 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:20 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:20 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:20 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:20 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:20 ntpd[13027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:20 ntpd[13027]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:20 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:20 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:20 ntpd[13027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:20 ntpd[13027]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:20.300 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:20 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:20.301 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:20 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:20.301 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:20 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:20.301 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:20 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:20.301 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:20 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:20.301 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:20 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:20.301 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:20 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:20.301 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:20 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:20.301 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:20 ntpd[12966]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:20.301 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:20 ntpd[12966]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:20.301 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:20 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:20.301 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:20 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:20.301 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:20 ntpd[12966]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:20.301 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:20 ntpd[12966]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:20.302 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:20 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:20.302 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:20 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:20.302 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:20 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:20.302 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:20 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:20.302 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:20 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:20.302 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:20 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:20.302 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:20 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:20.302 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:20 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:20.302 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:20 ntpd[12972]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:20.302 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:20 ntpd[12972]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:20.302 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:20 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:20.302 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:20 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:20.303 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:20 ntpd[12972]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:20.303 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:20 ntpd[12972]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:21.293 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:21 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:21.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:21 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:21.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:21 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:21.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:21 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:21.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:21 ntpd[13027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:21.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:21 ntpd[13027]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:21.299 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:21 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:21.299 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:21 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:21.299 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:21 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:21.299 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:21 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:21.299 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:21 ntpd[12966]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:21.299 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:21 ntpd[12966]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:22 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:22 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:22 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:22 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:22 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:22 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:22 ntpd[13027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:22 ntpd[13027]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:22 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:22 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:22 ntpd[13027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:22.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:22 ntpd[13027]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:22.299 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:22.299 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:22.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:22.301 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:22 ntpd[12972]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:22.302 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:22 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:22.302 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:22 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:22.302 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:22 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:22.302 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:22 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:22.302 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:22 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:22.302 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:22 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:22.302 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:22 ntpd[12966]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:22.302 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:22 ntpd[12966]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:22.302 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:22 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:22.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:22 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:22.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:22 ntpd[12966]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:22.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:22 ntpd[12966]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:23.293 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:23 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:23.293 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:23 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:23.293 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:23 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:23.293 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:23 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:23.293 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:23 ntpd[13027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:23.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:23 ntpd[13027]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:23.298 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:23 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:23.298 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:23 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:23.298 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:23 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:23.299 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:23 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:23.299 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:23 ntpd[12966]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:23.299 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:23 ntpd[12966]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:24.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:24 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:24 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:24 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:24 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:24 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:24 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:24 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:24 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:24 ntpd[13027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:24 ntpd[13027]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:24 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:24 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:24 ntpd[13027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:24.296 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:24 ntpd[13027]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:24.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:24.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:24.300 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.301 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.301 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:24.301 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:24.301 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.301 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.301 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:24.301 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:24 ntpd[12972]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:24.302 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:24 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:24 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:24 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:24 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:24 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:24 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:24 ntpd[12966]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:24.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:24 ntpd[12966]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:24.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:24 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:24.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:24 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:24.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:24 ntpd[12966]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:24.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:24 ntpd[12966]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:25.293 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:25 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:25.293 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:25 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:25.293 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:25 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:25 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:25 ntpd[13027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:25.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:25 ntpd[13027]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:25.298 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:25 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:25.299 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:25 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:25.299 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:25 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:25.299 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:25 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:25.299 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:25 ntpd[12966]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:25.299 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:25 ntpd[12966]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:26.294 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:26 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:26 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:26 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:26 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:26.295 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:26 ntpd[13027]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:26 ntpd[13027]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:26 ntpd[13027]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:26.296 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:26 ntpd[13027]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:26.296 INFO:teuthology.orchestra.run.smithi090.stdout:24 Apr 14:29:26 ntpd[13027]: ntpd: time slew +0.000993 s 2024-04-24T14:29:26.297 INFO:teuthology.orchestra.run.smithi090.stdout:ntpd: time slew +0.000993s 2024-04-24T14:29:26.297 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:26 ntpd[13027]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-24T14:29:26.298 INFO:teuthology.orchestra.run.smithi090.stderr:24 Apr 14:29:26 ntpd[13027]: can't open /var/log/ntpstats/loopstats.20240424: Permission denied 2024-04-24T14:29:26.302 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:26 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:26.302 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:26 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:26.302 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:26 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:26.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:26 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:26.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:26 ntpd[12966]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:26.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:26 ntpd[12966]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:26.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:26 ntpd[12966]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:26.303 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:26 ntpd[12966]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:26.304 INFO:teuthology.orchestra.run.smithi173.stdout:24 Apr 14:29:26 ntpd[12966]: ntpd: time slew +0.000246 s 2024-04-24T14:29:26.304 INFO:teuthology.orchestra.run.smithi173.stdout:ntpd: time slew +0.000246s 2024-04-24T14:29:26.306 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:26 ntpd[12966]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-24T14:29:26.306 INFO:teuthology.orchestra.run.smithi173.stderr:24 Apr 14:29:26 ntpd[12966]: can't open /var/log/ntpstats/loopstats.20240424: Permission denied 2024-04-24T14:29:26.308 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:26.308 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:26.309 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:26.309 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:26.309 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:26.309 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:26.309 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:26.309 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:26.309 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:26.309 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:26.309 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T14:29:26.309 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T14:29:26.310 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T14:29:26.310 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T14:29:26.310 INFO:teuthology.orchestra.run.smithi103.stdout:24 Apr 14:29:26 ntpd[12972]: ntpd: time slew +0.001312 s 2024-04-24T14:29:26.310 INFO:teuthology.orchestra.run.smithi103.stdout:ntpd: time slew +0.001312s 2024-04-24T14:29:26.311 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-24T14:29:26.311 INFO:teuthology.orchestra.run.smithi103.stderr:24 Apr 14:29:26 ntpd[12972]: can't open /var/log/ntpstats/loopstats.20240424: Permission denied 2024-04-24T14:29:26.363 INFO:teuthology.orchestra.run.smithi173.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T14:29:26.363 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================== 2024-04-24T14:29:26.363 INFO:teuthology.orchestra.run.smithi173.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:29:26.364 INFO:teuthology.orchestra.run.smithi173.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:29:26.364 INFO:teuthology.orchestra.run.smithi173.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:29:26.364 INFO:teuthology.orchestra.run.smithi173.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:29:26.367 INFO:teuthology.orchestra.run.smithi090.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T14:29:26.367 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================== 2024-04-24T14:29:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:29:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:29:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:29:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:29:26.390 INFO:teuthology.orchestra.run.smithi103.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T14:29:26.390 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================== 2024-04-24T14:29:26.390 INFO:teuthology.orchestra.run.smithi103.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:29:26.390 INFO:teuthology.orchestra.run.smithi103.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:29:26.390 INFO:teuthology.orchestra.run.smithi103.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:29:26.391 INFO:teuthology.orchestra.run.smithi103.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:29:26.391 INFO:teuthology.run_tasks:Running task install... 2024-04-24T14:29:26.409 DEBUG:teuthology.task.install:project ceph 2024-04-24T14:29:26.409 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}} 2024-04-24T14:29:26.409 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'} 2024-04-24T14:29:26.410 INFO:teuthology.task.install:Using flavor: default 2024-04-24T14:29:26.422 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T14:29:26.423 INFO:teuthology.task.install:extra packages: ['rbd-nbd'] 2024-04-24T14:29:26.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-key list | grep Ceph 2024-04-24T14:29:26.424 DEBUG:teuthology.orchestra.run.smithi103:> sudo apt-key list | grep Ceph 2024-04-24T14:29:26.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-key list | grep Ceph 2024-04-24T14:29:26.526 INFO:teuthology.orchestra.run.smithi173.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-04-24T14:29:26.530 INFO:teuthology.orchestra.run.smithi090.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-04-24T14:29:26.535 INFO:teuthology.orchestra.run.smithi103.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-04-24T14:29:26.584 INFO:teuthology.orchestra.run.smithi173.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-24T14:29:26.584 INFO:teuthology.orchestra.run.smithi173.stdout:uid [ unknown] Ceph.com (release key) 2024-04-24T14:29:26.585 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-04-24T14:29:26.585 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T14:29:26.590 INFO:teuthology.orchestra.run.smithi090.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-24T14:29:26.590 INFO:teuthology.orchestra.run.smithi090.stdout:uid [ unknown] Ceph.com (release key) 2024-04-24T14:29:26.590 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-04-24T14:29:26.590 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T14:29:26.594 INFO:teuthology.orchestra.run.smithi103.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-24T14:29:26.594 INFO:teuthology.orchestra.run.smithi103.stdout:uid [ unknown] Ceph.com (release key) 2024-04-24T14:29:26.595 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-04-24T14:29:26.596 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T14:29:26.742 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default/ 2024-04-24T14:29:26.743 INFO:teuthology.task.install.deb:Package version is 17.2.7-915-g330067db-1focal 2024-04-24T14:29:26.748 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default/ 2024-04-24T14:29:26.748 INFO:teuthology.task.install.deb:Package version is 17.2.7-915-g330067db-1focal 2024-04-24T14:29:26.819 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default/ 2024-04-24T14:29:26.819 INFO:teuthology.task.install.deb:Package version is 17.2.7-915-g330067db-1focal 2024-04-24T14:29:26.925 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:29:26.925 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-24T14:29:26.934 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-24T14:29:26.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-24T14:29:26.942 DEBUG:teuthology.orchestra.run.smithi103:> sudo apt-get update 2024-04-24T14:29:26.947 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:29:26.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-24T14:29:26.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-get update 2024-04-24T14:29:26.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get update 2024-04-24T14:29:27.103 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-04-24T14:29:27.112 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-04-24T14:29:27.135 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-04-24T14:29:27.173 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-04-24T14:29:27.174 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-04-24T14:29:27.195 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-04-24T14:29:27.222 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-04-24T14:29:27.224 INFO:teuthology.orchestra.run.smithi173.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal InRelease 2024-04-24T14:29:27.234 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-04-24T14:29:27.253 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-04-24T14:29:27.253 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-04-24T14:29:27.253 INFO:teuthology.orchestra.run.smithi103.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal InRelease 2024-04-24T14:29:27.262 INFO:teuthology.orchestra.run.smithi173.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-04-24T14:29:27.280 INFO:teuthology.orchestra.run.smithi090.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal InRelease 2024-04-24T14:29:27.290 INFO:teuthology.orchestra.run.smithi103.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-04-24T14:29:27.300 INFO:teuthology.orchestra.run.smithi173.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal Release.gpg 2024-04-24T14:29:27.318 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-04-24T14:29:27.322 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-04-24T14:29:27.328 INFO:teuthology.orchestra.run.smithi103.stdout:Ign:6 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal Release.gpg 2024-04-24T14:29:27.337 INFO:teuthology.orchestra.run.smithi173.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main i386 Packages [3,362 B] 2024-04-24T14:29:27.357 INFO:teuthology.orchestra.run.smithi090.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal Release.gpg 2024-04-24T14:29:27.365 INFO:teuthology.orchestra.run.smithi103.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-04-24T14:29:27.384 INFO:teuthology.orchestra.run.smithi173.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-04-24T14:29:27.394 INFO:teuthology.orchestra.run.smithi090.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-04-24T14:29:27.406 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-04-24T14:29:27.413 INFO:teuthology.orchestra.run.smithi103.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main i386 Packages [3,362 B] 2024-04-24T14:29:27.443 INFO:teuthology.orchestra.run.smithi090.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main i386 Packages [3,362 B] 2024-04-24T14:29:27.732 INFO:teuthology.orchestra.run.smithi173.stdout:Fetched 28.4 kB in 1s (45.8 kB/s) 2024-04-24T14:29:27.828 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 28.4 kB in 1s (44.7 kB/s) 2024-04-24T14:29:27.917 INFO:teuthology.orchestra.run.smithi103.stdout:Fetched 28.4 kB in 1s (37.1 kB/s) 2024-04-24T14:29:28.843 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2024-04-24T14:29:28.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-915-g330067db-1focal cephadm=17.2.7-915-g330067db-1focal ceph-mds=17.2.7-915-g330067db-1focal ceph-mgr=17.2.7-915-g330067db-1focal ceph-common=17.2.7-915-g330067db-1focal ceph-fuse=17.2.7-915-g330067db-1focal ceph-test=17.2.7-915-g330067db-1focal ceph-volume=17.2.7-915-g330067db-1focal radosgw=17.2.7-915-g330067db-1focal python3-rados=17.2.7-915-g330067db-1focal python3-rgw=17.2.7-915-g330067db-1focal python3-cephfs=17.2.7-915-g330067db-1focal python3-rbd=17.2.7-915-g330067db-1focal libcephfs2=17.2.7-915-g330067db-1focal libcephfs-dev=17.2.7-915-g330067db-1focal librados2=17.2.7-915-g330067db-1focal librbd1=17.2.7-915-g330067db-1focal rbd-fuse=17.2.7-915-g330067db-1focal rbd-nbd=17.2.7-915-g330067db-1focal 2024-04-24T14:29:28.935 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2024-04-24T14:29:28.940 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2024-04-24T14:29:28.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-915-g330067db-1focal cephadm=17.2.7-915-g330067db-1focal ceph-mds=17.2.7-915-g330067db-1focal ceph-mgr=17.2.7-915-g330067db-1focal ceph-common=17.2.7-915-g330067db-1focal ceph-fuse=17.2.7-915-g330067db-1focal ceph-test=17.2.7-915-g330067db-1focal ceph-volume=17.2.7-915-g330067db-1focal radosgw=17.2.7-915-g330067db-1focal python3-rados=17.2.7-915-g330067db-1focal python3-rgw=17.2.7-915-g330067db-1focal python3-cephfs=17.2.7-915-g330067db-1focal python3-rbd=17.2.7-915-g330067db-1focal libcephfs2=17.2.7-915-g330067db-1focal libcephfs-dev=17.2.7-915-g330067db-1focal librados2=17.2.7-915-g330067db-1focal librbd1=17.2.7-915-g330067db-1focal rbd-fuse=17.2.7-915-g330067db-1focal rbd-nbd=17.2.7-915-g330067db-1focal 2024-04-24T14:29:29.014 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2024-04-24T14:29:29.030 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2024-04-24T14:29:29.040 DEBUG:teuthology.orchestra.run.smithi103:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-915-g330067db-1focal cephadm=17.2.7-915-g330067db-1focal ceph-mds=17.2.7-915-g330067db-1focal ceph-mgr=17.2.7-915-g330067db-1focal ceph-common=17.2.7-915-g330067db-1focal ceph-fuse=17.2.7-915-g330067db-1focal ceph-test=17.2.7-915-g330067db-1focal ceph-volume=17.2.7-915-g330067db-1focal radosgw=17.2.7-915-g330067db-1focal python3-rados=17.2.7-915-g330067db-1focal python3-rgw=17.2.7-915-g330067db-1focal python3-cephfs=17.2.7-915-g330067db-1focal python3-rbd=17.2.7-915-g330067db-1focal libcephfs2=17.2.7-915-g330067db-1focal libcephfs-dev=17.2.7-915-g330067db-1focal librados2=17.2.7-915-g330067db-1focal librbd1=17.2.7-915-g330067db-1focal rbd-fuse=17.2.7-915-g330067db-1focal rbd-nbd=17.2.7-915-g330067db-1focal 2024-04-24T14:29:29.114 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2024-04-24T14:29:29.150 INFO:teuthology.orchestra.run.smithi173.stdout:Building dependency tree... 2024-04-24T14:29:29.151 INFO:teuthology.orchestra.run.smithi173.stdout:Reading state information... 2024-04-24T14:29:29.219 INFO:teuthology.orchestra.run.smithi090.stdout:Building dependency tree... 2024-04-24T14:29:29.219 INFO:teuthology.orchestra.run.smithi090.stdout:Reading state information... 2024-04-24T14:29:29.326 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2024-04-24T14:29:29.327 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2024-04-24T14:29:29.334 INFO:teuthology.orchestra.run.smithi173.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T14:29:29.335 INFO:teuthology.orchestra.run.smithi173.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-04-24T14:29:29.335 INFO:teuthology.orchestra.run.smithi173.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T14:29:29.335 INFO:teuthology.orchestra.run.smithi173.stdout:The following additional packages will be installed: 2024-04-24T14:29:29.335 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-24T14:29:29.336 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-04-24T14:29:29.336 INFO:teuthology.orchestra.run.smithi173.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-04-24T14:29:29.336 INFO:teuthology.orchestra.run.smithi173.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-04-24T14:29:29.336 INFO:teuthology.orchestra.run.smithi173.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-04-24T14:29:29.336 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-04-24T14:29:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-04-24T14:29:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-04-24T14:29:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-04-24T14:29:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-04-24T14:29:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-04-24T14:29:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-04-24T14:29:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-04-24T14:29:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-24T14:29:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-04-24T14:29:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-04-24T14:29:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-04-24T14:29:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-24T14:29:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-04-24T14:29:29.337 INFO:teuthology.orchestra.run.smithi173.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-04-24T14:29:29.338 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-04-24T14:29:29.338 INFO:teuthology.orchestra.run.smithi173.stdout: socat xmlstarlet 2024-04-24T14:29:29.338 INFO:teuthology.orchestra.run.smithi173.stdout:Suggested packages: 2024-04-24T14:29:29.338 INFO:teuthology.orchestra.run.smithi173.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-04-24T14:29:29.338 INFO:teuthology.orchestra.run.smithi173.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-04-24T14:29:29.338 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-04-24T14:29:29.338 INFO:teuthology.orchestra.run.smithi173.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-04-24T14:29:29.338 INFO:teuthology.orchestra.run.smithi173.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-04-24T14:29:29.339 INFO:teuthology.orchestra.run.smithi173.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-04-24T14:29:29.339 INFO:teuthology.orchestra.run.smithi173.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-04-24T14:29:29.339 INFO:teuthology.orchestra.run.smithi173.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-24T14:29:29.339 INFO:teuthology.orchestra.run.smithi173.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-04-24T14:29:29.339 INFO:teuthology.orchestra.run.smithi173.stdout:Recommended packages: 2024-04-24T14:29:29.339 INFO:teuthology.orchestra.run.smithi173.stdout: btrfs-tools 2024-04-24T14:29:29.401 INFO:teuthology.orchestra.run.smithi090.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T14:29:29.402 INFO:teuthology.orchestra.run.smithi090.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-04-24T14:29:29.402 INFO:teuthology.orchestra.run.smithi090.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T14:29:29.402 INFO:teuthology.orchestra.run.smithi090.stdout:The following additional packages will be installed: 2024-04-24T14:29:29.403 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-24T14:29:29.403 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-04-24T14:29:29.404 INFO:teuthology.orchestra.run.smithi090.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-04-24T14:29:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-04-24T14:29:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-04-24T14:29:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-04-24T14:29:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-04-24T14:29:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-04-24T14:29:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-04-24T14:29:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-04-24T14:29:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-04-24T14:29:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-04-24T14:29:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-04-24T14:29:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-24T14:29:29.405 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-04-24T14:29:29.406 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-04-24T14:29:29.406 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-04-24T14:29:29.406 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-24T14:29:29.406 INFO:teuthology.orchestra.run.smithi090.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-04-24T14:29:29.406 INFO:teuthology.orchestra.run.smithi090.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-04-24T14:29:29.406 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-04-24T14:29:29.406 INFO:teuthology.orchestra.run.smithi090.stdout: socat xmlstarlet 2024-04-24T14:29:29.407 INFO:teuthology.orchestra.run.smithi090.stdout:Suggested packages: 2024-04-24T14:29:29.407 INFO:teuthology.orchestra.run.smithi090.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-04-24T14:29:29.407 INFO:teuthology.orchestra.run.smithi090.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-04-24T14:29:29.407 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-04-24T14:29:29.407 INFO:teuthology.orchestra.run.smithi090.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-04-24T14:29:29.407 INFO:teuthology.orchestra.run.smithi090.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-04-24T14:29:29.407 INFO:teuthology.orchestra.run.smithi090.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-04-24T14:29:29.408 INFO:teuthology.orchestra.run.smithi090.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-04-24T14:29:29.408 INFO:teuthology.orchestra.run.smithi090.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-24T14:29:29.408 INFO:teuthology.orchestra.run.smithi090.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-04-24T14:29:29.408 INFO:teuthology.orchestra.run.smithi090.stdout:Recommended packages: 2024-04-24T14:29:29.408 INFO:teuthology.orchestra.run.smithi090.stdout: btrfs-tools 2024-04-24T14:29:29.458 INFO:teuthology.orchestra.run.smithi173.stdout:The following NEW packages will be installed: 2024-04-24T14:29:29.458 INFO:teuthology.orchestra.run.smithi173.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-24T14:29:29.458 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-24T14:29:29.458 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-24T14:29:29.458 INFO:teuthology.orchestra.run.smithi173.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-04-24T14:29:29.458 INFO:teuthology.orchestra.run.smithi173.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-04-24T14:29:29.458 INFO:teuthology.orchestra.run.smithi173.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-24T14:29:29.459 INFO:teuthology.orchestra.run.smithi173.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-04-24T14:29:29.459 INFO:teuthology.orchestra.run.smithi173.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-04-24T14:29:29.459 INFO:teuthology.orchestra.run.smithi173.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-04-24T14:29:29.459 INFO:teuthology.orchestra.run.smithi173.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-04-24T14:29:29.460 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-04-24T14:29:29.460 INFO:teuthology.orchestra.run.smithi173.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-04-24T14:29:29.460 INFO:teuthology.orchestra.run.smithi173.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-04-24T14:29:29.460 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-04-24T14:29:29.460 INFO:teuthology.orchestra.run.smithi173.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-04-24T14:29:29.460 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-04-24T14:29:29.460 INFO:teuthology.orchestra.run.smithi173.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T14:29:29.460 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-24T14:29:29.460 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-04-24T14:29:29.460 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-04-24T14:29:29.460 INFO:teuthology.orchestra.run.smithi173.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-04-24T14:29:29.460 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-04-24T14:29:29.460 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-04-24T14:29:29.460 INFO:teuthology.orchestra.run.smithi173.stdout: qttranslations5-l10n radosgw rbd-fuse rbd-nbd socat xmlstarlet 2024-04-24T14:29:29.461 INFO:teuthology.orchestra.run.smithi173.stdout:The following packages will be upgraded: 2024-04-24T14:29:29.461 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 librbd1 2024-04-24T14:29:29.514 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T14:29:29.516 INFO:teuthology.orchestra.run.smithi103.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-04-24T14:29:29.516 INFO:teuthology.orchestra.run.smithi103.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T14:29:29.516 INFO:teuthology.orchestra.run.smithi103.stdout:The following additional packages will be installed: 2024-04-24T14:29:29.516 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-24T14:29:29.516 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-04-24T14:29:29.516 INFO:teuthology.orchestra.run.smithi103.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-04-24T14:29:29.516 INFO:teuthology.orchestra.run.smithi103.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-04-24T14:29:29.516 INFO:teuthology.orchestra.run.smithi103.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-04-24T14:29:29.517 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-04-24T14:29:29.517 INFO:teuthology.orchestra.run.smithi103.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-04-24T14:29:29.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-04-24T14:29:29.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-04-24T14:29:29.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-04-24T14:29:29.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-04-24T14:29:29.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-04-24T14:29:29.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-04-24T14:29:29.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-24T14:29:29.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-04-24T14:29:29.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-04-24T14:29:29.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-04-24T14:29:29.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-24T14:29:29.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-04-24T14:29:29.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-04-24T14:29:29.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-04-24T14:29:29.518 INFO:teuthology.orchestra.run.smithi103.stdout: socat xmlstarlet 2024-04-24T14:29:29.519 INFO:teuthology.orchestra.run.smithi103.stdout:Suggested packages: 2024-04-24T14:29:29.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-04-24T14:29:29.519 INFO:teuthology.orchestra.run.smithi103.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-04-24T14:29:29.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-04-24T14:29:29.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-04-24T14:29:29.519 INFO:teuthology.orchestra.run.smithi103.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-04-24T14:29:29.519 INFO:teuthology.orchestra.run.smithi103.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-04-24T14:29:29.519 INFO:teuthology.orchestra.run.smithi103.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-04-24T14:29:29.519 INFO:teuthology.orchestra.run.smithi103.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-24T14:29:29.519 INFO:teuthology.orchestra.run.smithi103.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-04-24T14:29:29.519 INFO:teuthology.orchestra.run.smithi103.stdout:Recommended packages: 2024-04-24T14:29:29.520 INFO:teuthology.orchestra.run.smithi103.stdout: btrfs-tools 2024-04-24T14:29:29.527 INFO:teuthology.orchestra.run.smithi090.stdout:The following NEW packages will be installed: 2024-04-24T14:29:29.527 INFO:teuthology.orchestra.run.smithi090.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-24T14:29:29.527 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-24T14:29:29.527 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-24T14:29:29.529 INFO:teuthology.orchestra.run.smithi090.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-04-24T14:29:29.529 INFO:teuthology.orchestra.run.smithi090.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-04-24T14:29:29.529 INFO:teuthology.orchestra.run.smithi090.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-24T14:29:29.529 INFO:teuthology.orchestra.run.smithi090.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-04-24T14:29:29.529 INFO:teuthology.orchestra.run.smithi090.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-04-24T14:29:29.529 INFO:teuthology.orchestra.run.smithi090.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-04-24T14:29:29.529 INFO:teuthology.orchestra.run.smithi090.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-04-24T14:29:29.529 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-04-24T14:29:29.530 INFO:teuthology.orchestra.run.smithi090.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-04-24T14:29:29.530 INFO:teuthology.orchestra.run.smithi090.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-04-24T14:29:29.530 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-04-24T14:29:29.530 INFO:teuthology.orchestra.run.smithi090.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-04-24T14:29:29.530 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-04-24T14:29:29.530 INFO:teuthology.orchestra.run.smithi090.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T14:29:29.530 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-24T14:29:29.530 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-04-24T14:29:29.530 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-04-24T14:29:29.530 INFO:teuthology.orchestra.run.smithi090.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-04-24T14:29:29.530 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-04-24T14:29:29.530 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-04-24T14:29:29.530 INFO:teuthology.orchestra.run.smithi090.stdout: qttranslations5-l10n radosgw rbd-fuse rbd-nbd socat xmlstarlet 2024-04-24T14:29:29.531 INFO:teuthology.orchestra.run.smithi090.stdout:The following packages will be upgraded: 2024-04-24T14:29:29.531 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 librbd1 2024-04-24T14:29:29.533 INFO:teuthology.orchestra.run.smithi173.stdout:2 upgraded, 109 newly installed, 0 to remove and 277 not upgraded. 2024-04-24T14:29:29.533 INFO:teuthology.orchestra.run.smithi173.stdout:Need to get 201 MB of archives. 2024-04-24T14:29:29.533 INFO:teuthology.orchestra.run.smithi173.stdout:After this operation, 936 MB of additional disk space will be used. 2024-04-24T14:29:29.533 INFO:teuthology.orchestra.run.smithi173.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-04-24T14:29:29.601 INFO:teuthology.orchestra.run.smithi090.stdout:2 upgraded, 109 newly installed, 0 to remove and 277 not upgraded. 2024-04-24T14:29:29.601 INFO:teuthology.orchestra.run.smithi090.stdout:Need to get 201 MB of archives. 2024-04-24T14:29:29.601 INFO:teuthology.orchestra.run.smithi090.stdout:After this operation, 936 MB of additional disk space will be used. 2024-04-24T14:29:29.601 INFO:teuthology.orchestra.run.smithi090.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-04-24T14:29:29.608 INFO:teuthology.orchestra.run.smithi173.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-915-g330067db-1focal [3,504 kB] 2024-04-24T14:29:29.630 INFO:teuthology.orchestra.run.smithi173.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-04-24T14:29:29.631 INFO:teuthology.orchestra.run.smithi173.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-04-24T14:29:29.637 INFO:teuthology.orchestra.run.smithi103.stdout:The following NEW packages will be installed: 2024-04-24T14:29:29.637 INFO:teuthology.orchestra.run.smithi103.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-24T14:29:29.637 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-24T14:29:29.637 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-24T14:29:29.638 INFO:teuthology.orchestra.run.smithi103.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-04-24T14:29:29.638 INFO:teuthology.orchestra.run.smithi103.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-04-24T14:29:29.638 INFO:teuthology.orchestra.run.smithi103.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-24T14:29:29.638 INFO:teuthology.orchestra.run.smithi103.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-04-24T14:29:29.639 INFO:teuthology.orchestra.run.smithi103.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-04-24T14:29:29.639 INFO:teuthology.orchestra.run.smithi103.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-04-24T14:29:29.639 INFO:teuthology.orchestra.run.smithi103.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-04-24T14:29:29.639 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-04-24T14:29:29.639 INFO:teuthology.orchestra.run.smithi103.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-04-24T14:29:29.639 INFO:teuthology.orchestra.run.smithi103.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-04-24T14:29:29.639 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-04-24T14:29:29.639 INFO:teuthology.orchestra.run.smithi103.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-04-24T14:29:29.639 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-04-24T14:29:29.639 INFO:teuthology.orchestra.run.smithi103.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T14:29:29.639 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-24T14:29:29.640 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-04-24T14:29:29.640 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-04-24T14:29:29.640 INFO:teuthology.orchestra.run.smithi103.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-04-24T14:29:29.640 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-04-24T14:29:29.640 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-04-24T14:29:29.640 INFO:teuthology.orchestra.run.smithi103.stdout: qttranslations5-l10n radosgw rbd-fuse rbd-nbd socat xmlstarlet 2024-04-24T14:29:29.640 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages will be upgraded: 2024-04-24T14:29:29.640 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 librbd1 2024-04-24T14:29:29.641 INFO:teuthology.orchestra.run.smithi173.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-04-24T14:29:29.649 INFO:teuthology.orchestra.run.smithi173.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-04-24T14:29:29.677 INFO:teuthology.orchestra.run.smithi173.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-04-24T14:29:29.678 INFO:teuthology.orchestra.run.smithi173.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-04-24T14:29:29.685 INFO:teuthology.orchestra.run.smithi173.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-04-24T14:29:29.698 INFO:teuthology.orchestra.run.smithi090.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-04-24T14:29:29.709 INFO:teuthology.orchestra.run.smithi090.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-04-24T14:29:29.722 INFO:teuthology.orchestra.run.smithi090.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-04-24T14:29:29.727 INFO:teuthology.orchestra.run.smithi173.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-04-24T14:29:29.727 INFO:teuthology.orchestra.run.smithi090.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-04-24T14:29:29.730 INFO:teuthology.orchestra.run.smithi173.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-04-24T14:29:29.742 INFO:teuthology.orchestra.run.smithi173.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-04-24T14:29:29.744 INFO:teuthology.orchestra.run.smithi173.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-04-24T14:29:29.744 INFO:teuthology.orchestra.run.smithi173.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-04-24T14:29:29.747 INFO:teuthology.orchestra.run.smithi173.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-04-24T14:29:29.747 INFO:teuthology.orchestra.run.smithi173.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-04-24T14:29:29.748 INFO:teuthology.orchestra.run.smithi173.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-04-24T14:29:29.750 INFO:teuthology.orchestra.run.smithi173.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-04-24T14:29:29.756 INFO:teuthology.orchestra.run.smithi173.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-04-24T14:29:29.767 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-04-24T14:29:29.769 INFO:teuthology.orchestra.run.smithi090.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-04-24T14:29:29.771 INFO:teuthology.orchestra.run.smithi173.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-04-24T14:29:29.771 INFO:teuthology.orchestra.run.smithi173.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-04-24T14:29:29.771 INFO:teuthology.orchestra.run.smithi173.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-04-24T14:29:29.773 INFO:teuthology.orchestra.run.smithi173.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-24T14:29:29.773 INFO:teuthology.orchestra.run.smithi173.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-04-24T14:29:29.774 INFO:teuthology.orchestra.run.smithi173.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2024-04-24T14:29:29.775 INFO:teuthology.orchestra.run.smithi173.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-04-24T14:29:29.776 INFO:teuthology.orchestra.run.smithi090.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-915-g330067db-1focal [3,504 kB] 2024-04-24T14:29:29.776 INFO:teuthology.orchestra.run.smithi173.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-04-24T14:29:29.778 INFO:teuthology.orchestra.run.smithi090.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-04-24T14:29:29.778 INFO:teuthology.orchestra.run.smithi173.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-04-24T14:29:29.798 INFO:teuthology.orchestra.run.smithi173.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-04-24T14:29:29.799 INFO:teuthology.orchestra.run.smithi173.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-04-24T14:29:29.805 INFO:teuthology.orchestra.run.smithi173.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-04-24T14:29:29.806 INFO:teuthology.orchestra.run.smithi173.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-04-24T14:29:29.807 INFO:teuthology.orchestra.run.smithi173.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-04-24T14:29:29.808 INFO:teuthology.orchestra.run.smithi173.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-04-24T14:29:29.810 INFO:teuthology.orchestra.run.smithi173.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2024-04-24T14:29:29.812 INFO:teuthology.orchestra.run.smithi173.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.2 [95.9 kB] 2024-04-24T14:29:29.814 INFO:teuthology.orchestra.run.smithi173.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-915-g330067db-1focal [3,558 kB] 2024-04-24T14:29:29.826 INFO:teuthology.orchestra.run.smithi173.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-24T14:29:29.827 INFO:teuthology.orchestra.run.smithi173.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-04-24T14:29:29.828 INFO:teuthology.orchestra.run.smithi090.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-04-24T14:29:29.831 INFO:teuthology.orchestra.run.smithi090.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-04-24T14:29:29.834 INFO:teuthology.orchestra.run.smithi103.stdout:2 upgraded, 109 newly installed, 0 to remove and 277 not upgraded. 2024-04-24T14:29:29.834 INFO:teuthology.orchestra.run.smithi103.stdout:Need to get 201 MB of archives. 2024-04-24T14:29:29.834 INFO:teuthology.orchestra.run.smithi103.stdout:After this operation, 936 MB of additional disk space will be used. 2024-04-24T14:29:29.834 INFO:teuthology.orchestra.run.smithi103.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-04-24T14:29:29.846 INFO:teuthology.orchestra.run.smithi090.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-04-24T14:29:29.852 INFO:teuthology.orchestra.run.smithi090.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-04-24T14:29:29.853 INFO:teuthology.orchestra.run.smithi090.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-04-24T14:29:29.855 INFO:teuthology.orchestra.run.smithi090.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-04-24T14:29:29.856 INFO:teuthology.orchestra.run.smithi090.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-04-24T14:29:29.857 INFO:teuthology.orchestra.run.smithi090.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-04-24T14:29:29.859 INFO:teuthology.orchestra.run.smithi090.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-04-24T14:29:29.861 INFO:teuthology.orchestra.run.smithi173.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-04-24T14:29:29.863 INFO:teuthology.orchestra.run.smithi173.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-04-24T14:29:29.865 INFO:teuthology.orchestra.run.smithi173.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-04-24T14:29:29.866 INFO:teuthology.orchestra.run.smithi173.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-04-24T14:29:29.868 INFO:teuthology.orchestra.run.smithi090.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-04-24T14:29:29.869 INFO:teuthology.orchestra.run.smithi173.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-04-24T14:29:29.903 INFO:teuthology.orchestra.run.smithi090.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-04-24T14:29:29.904 INFO:teuthology.orchestra.run.smithi090.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-04-24T14:29:29.904 INFO:teuthology.orchestra.run.smithi090.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-04-24T14:29:29.906 INFO:teuthology.orchestra.run.smithi090.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-24T14:29:29.907 INFO:teuthology.orchestra.run.smithi090.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-04-24T14:29:29.907 INFO:teuthology.orchestra.run.smithi090.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2024-04-24T14:29:29.910 INFO:teuthology.orchestra.run.smithi090.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-04-24T14:29:29.911 INFO:teuthology.orchestra.run.smithi090.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-04-24T14:29:29.913 INFO:teuthology.orchestra.run.smithi090.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-04-24T14:29:29.924 INFO:teuthology.orchestra.run.smithi173.stdout:Get:45 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-915-g330067db-1focal [761 kB] 2024-04-24T14:29:29.951 INFO:teuthology.orchestra.run.smithi173.stdout:Get:46 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-915-g330067db-1focal [412 kB] 2024-04-24T14:29:29.958 INFO:teuthology.orchestra.run.smithi090.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-04-24T14:29:29.958 INFO:teuthology.orchestra.run.smithi090.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-04-24T14:29:29.965 INFO:teuthology.orchestra.run.smithi173.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-915-g330067db-1focal [22.6 kB] 2024-04-24T14:29:29.965 INFO:teuthology.orchestra.run.smithi173.stdout:Get:48 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-915-g330067db-1focal [200 kB] 2024-04-24T14:29:29.967 INFO:teuthology.orchestra.run.smithi090.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-04-24T14:29:29.968 INFO:teuthology.orchestra.run.smithi090.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-04-24T14:29:29.969 INFO:teuthology.orchestra.run.smithi090.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-04-24T14:29:29.972 INFO:teuthology.orchestra.run.smithi173.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-915-g330067db-1focal [47.8 kB] 2024-04-24T14:29:29.973 INFO:teuthology.orchestra.run.smithi173.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-915-g330067db-1focal [386 kB] 2024-04-24T14:29:29.987 INFO:teuthology.orchestra.run.smithi173.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-915-g330067db-1focal [5,601 kB] 2024-04-24T14:29:29.996 INFO:teuthology.orchestra.run.smithi103.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-915-g330067db-1focal [3,504 kB] 2024-04-24T14:29:30.009 INFO:teuthology.orchestra.run.smithi090.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-04-24T14:29:30.010 INFO:teuthology.orchestra.run.smithi090.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2024-04-24T14:29:30.013 INFO:teuthology.orchestra.run.smithi090.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.2 [95.9 kB] 2024-04-24T14:29:30.015 INFO:teuthology.orchestra.run.smithi090.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-24T14:29:30.015 INFO:teuthology.orchestra.run.smithi090.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-04-24T14:29:30.048 INFO:teuthology.orchestra.run.smithi173.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-04-24T14:29:30.069 INFO:teuthology.orchestra.run.smithi090.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-04-24T14:29:30.070 INFO:teuthology.orchestra.run.smithi090.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-04-24T14:29:30.072 INFO:teuthology.orchestra.run.smithi090.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-04-24T14:29:30.073 INFO:teuthology.orchestra.run.smithi090.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-04-24T14:29:30.074 INFO:teuthology.orchestra.run.smithi090.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-04-24T14:29:30.159 INFO:teuthology.orchestra.run.smithi173.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-04-24T14:29:30.160 INFO:teuthology.orchestra.run.smithi173.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-04-24T14:29:30.172 INFO:teuthology.orchestra.run.smithi173.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-04-24T14:29:30.176 INFO:teuthology.orchestra.run.smithi173.stdout:Get:56 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-915-g330067db-1focal [116 kB] 2024-04-24T14:29:30.177 INFO:teuthology.orchestra.run.smithi173.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-04-24T14:29:30.179 INFO:teuthology.orchestra.run.smithi173.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-915-g330067db-1focal [479 kB] 2024-04-24T14:29:30.192 INFO:teuthology.orchestra.run.smithi173.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-04-24T14:29:30.194 INFO:teuthology.orchestra.run.smithi173.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-04-24T14:29:30.194 INFO:teuthology.orchestra.run.smithi173.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-04-24T14:29:30.195 INFO:teuthology.orchestra.run.smithi173.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-915-g330067db-1focal [26.9 MB] 2024-04-24T14:29:30.196 INFO:teuthology.orchestra.run.smithi173.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-04-24T14:29:30.197 INFO:teuthology.orchestra.run.smithi173.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-04-24T14:29:30.202 INFO:teuthology.orchestra.run.smithi173.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-04-24T14:29:30.238 INFO:teuthology.orchestra.run.smithi103.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-04-24T14:29:30.246 INFO:teuthology.orchestra.run.smithi103.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-04-24T14:29:30.246 INFO:teuthology.orchestra.run.smithi173.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-04-24T14:29:30.247 INFO:teuthology.orchestra.run.smithi173.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-04-24T14:29:30.252 INFO:teuthology.orchestra.run.smithi173.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-04-24T14:29:30.254 INFO:teuthology.orchestra.run.smithi173.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-04-24T14:29:30.257 INFO:teuthology.orchestra.run.smithi173.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-04-24T14:29:30.258 INFO:teuthology.orchestra.run.smithi173.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-04-24T14:29:30.263 INFO:teuthology.orchestra.run.smithi173.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-04-24T14:29:30.265 INFO:teuthology.orchestra.run.smithi173.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-04-24T14:29:30.289 INFO:teuthology.orchestra.run.smithi103.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-04-24T14:29:30.320 INFO:teuthology.orchestra.run.smithi103.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-04-24T14:29:30.362 INFO:teuthology.orchestra.run.smithi173.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2024-04-24T14:29:30.365 INFO:teuthology.orchestra.run.smithi173.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-04-24T14:29:30.373 INFO:teuthology.orchestra.run.smithi090.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-04-24T14:29:30.429 INFO:teuthology.orchestra.run.smithi103.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-04-24T14:29:30.435 INFO:teuthology.orchestra.run.smithi103.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-04-24T14:29:30.440 INFO:teuthology.orchestra.run.smithi090.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-04-24T14:29:30.441 INFO:teuthology.orchestra.run.smithi173.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-04-24T14:29:30.443 INFO:teuthology.orchestra.run.smithi173.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-04-24T14:29:30.443 INFO:teuthology.orchestra.run.smithi173.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-04-24T14:29:30.446 INFO:teuthology.orchestra.run.smithi090.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-04-24T14:29:30.446 INFO:teuthology.orchestra.run.smithi173.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-04-24T14:29:30.446 INFO:teuthology.orchestra.run.smithi173.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-04-24T14:29:30.453 INFO:teuthology.orchestra.run.smithi173.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-04-24T14:29:30.465 INFO:teuthology.orchestra.run.smithi103.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-04-24T14:29:30.479 INFO:teuthology.orchestra.run.smithi090.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-04-24T14:29:30.480 INFO:teuthology.orchestra.run.smithi090.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-04-24T14:29:30.480 INFO:teuthology.orchestra.run.smithi173.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-04-24T14:29:30.480 INFO:teuthology.orchestra.run.smithi173.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-24T14:29:30.481 INFO:teuthology.orchestra.run.smithi090.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-04-24T14:29:30.481 INFO:teuthology.orchestra.run.smithi090.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-04-24T14:29:30.481 INFO:teuthology.orchestra.run.smithi173.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-04-24T14:29:30.482 INFO:teuthology.orchestra.run.smithi090.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-04-24T14:29:30.483 INFO:teuthology.orchestra.run.smithi090.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-04-24T14:29:30.483 INFO:teuthology.orchestra.run.smithi090.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-04-24T14:29:30.490 INFO:teuthology.orchestra.run.smithi090.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-04-24T14:29:30.495 INFO:teuthology.orchestra.run.smithi173.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-04-24T14:29:30.496 INFO:teuthology.orchestra.run.smithi173.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-04-24T14:29:30.497 INFO:teuthology.orchestra.run.smithi173.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-04-24T14:29:30.498 INFO:teuthology.orchestra.run.smithi173.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-04-24T14:29:30.498 INFO:teuthology.orchestra.run.smithi173.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-04-24T14:29:30.500 INFO:teuthology.orchestra.run.smithi173.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-04-24T14:29:30.508 INFO:teuthology.orchestra.run.smithi090.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-04-24T14:29:30.509 INFO:teuthology.orchestra.run.smithi090.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-04-24T14:29:30.514 INFO:teuthology.orchestra.run.smithi090.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-04-24T14:29:30.516 INFO:teuthology.orchestra.run.smithi090.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-04-24T14:29:30.518 INFO:teuthology.orchestra.run.smithi090.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-04-24T14:29:30.519 INFO:teuthology.orchestra.run.smithi090.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-04-24T14:29:30.520 INFO:teuthology.orchestra.run.smithi173.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-04-24T14:29:30.524 INFO:teuthology.orchestra.run.smithi090.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-04-24T14:29:30.532 INFO:teuthology.orchestra.run.smithi090.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-04-24T14:29:30.574 INFO:teuthology.orchestra.run.smithi090.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2024-04-24T14:29:30.578 INFO:teuthology.orchestra.run.smithi090.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-04-24T14:29:30.607 INFO:teuthology.orchestra.run.smithi103.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-04-24T14:29:30.615 INFO:teuthology.orchestra.run.smithi103.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-04-24T14:29:30.638 INFO:teuthology.orchestra.run.smithi103.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-04-24T14:29:30.661 INFO:teuthology.orchestra.run.smithi090.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-04-24T14:29:30.662 INFO:teuthology.orchestra.run.smithi090.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-04-24T14:29:30.662 INFO:teuthology.orchestra.run.smithi090.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-04-24T14:29:30.662 INFO:teuthology.orchestra.run.smithi090.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-04-24T14:29:30.663 INFO:teuthology.orchestra.run.smithi090.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-04-24T14:29:30.664 INFO:teuthology.orchestra.run.smithi090.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-04-24T14:29:30.686 INFO:teuthology.orchestra.run.smithi090.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-04-24T14:29:30.686 INFO:teuthology.orchestra.run.smithi090.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-24T14:29:30.687 INFO:teuthology.orchestra.run.smithi090.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-04-24T14:29:30.691 INFO:teuthology.orchestra.run.smithi090.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-04-24T14:29:30.712 INFO:teuthology.orchestra.run.smithi090.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-04-24T14:29:30.713 INFO:teuthology.orchestra.run.smithi090.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-04-24T14:29:30.713 INFO:teuthology.orchestra.run.smithi090.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-04-24T14:29:30.714 INFO:teuthology.orchestra.run.smithi090.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-04-24T14:29:30.716 INFO:teuthology.orchestra.run.smithi090.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-04-24T14:29:30.717 INFO:teuthology.orchestra.run.smithi090.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-04-24T14:29:30.809 INFO:teuthology.orchestra.run.smithi103.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-04-24T14:29:31.062 INFO:teuthology.orchestra.run.smithi103.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-04-24T14:29:31.215 INFO:teuthology.orchestra.run.smithi173.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-915-g330067db-1focal [5,170 kB] 2024-04-24T14:29:31.260 INFO:teuthology.orchestra.run.smithi103.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-04-24T14:29:31.281 INFO:teuthology.orchestra.run.smithi103.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-04-24T14:29:31.288 INFO:teuthology.orchestra.run.smithi173.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-915-g330067db-1focal [223 kB] 2024-04-24T14:29:31.295 INFO:teuthology.orchestra.run.smithi103.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-04-24T14:29:31.296 INFO:teuthology.orchestra.run.smithi173.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-915-g330067db-1focal [158 kB] 2024-04-24T14:29:31.323 INFO:teuthology.orchestra.run.smithi173.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-915-g330067db-1focal [1,594 kB] 2024-04-24T14:29:31.347 INFO:teuthology.orchestra.run.smithi103.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-04-24T14:29:31.354 INFO:teuthology.orchestra.run.smithi103.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-04-24T14:29:31.357 INFO:teuthology.orchestra.run.smithi173.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-915-g330067db-1focal [4,686 kB] 2024-04-24T14:29:31.358 INFO:teuthology.orchestra.run.smithi103.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-04-24T14:29:31.361 INFO:teuthology.orchestra.run.smithi103.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-04-24T14:29:31.364 INFO:teuthology.orchestra.run.smithi103.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-04-24T14:29:31.376 INFO:teuthology.orchestra.run.smithi103.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-24T14:29:31.516 INFO:teuthology.orchestra.run.smithi173.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-915-g330067db-1focal [23.1 MB] 2024-04-24T14:29:31.542 INFO:teuthology.orchestra.run.smithi103.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-04-24T14:29:31.626 INFO:teuthology.orchestra.run.smithi103.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2024-04-24T14:29:31.831 INFO:teuthology.orchestra.run.smithi103.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-04-24T14:29:31.871 INFO:teuthology.orchestra.run.smithi103.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-04-24T14:29:31.936 INFO:teuthology.orchestra.run.smithi103.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-04-24T14:29:31.977 INFO:teuthology.orchestra.run.smithi103.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-04-24T14:29:31.984 INFO:teuthology.orchestra.run.smithi103.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-04-24T14:29:32.095 INFO:teuthology.orchestra.run.smithi103.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-04-24T14:29:32.096 INFO:teuthology.orchestra.run.smithi103.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-04-24T14:29:32.100 INFO:teuthology.orchestra.run.smithi103.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-04-24T14:29:32.103 INFO:teuthology.orchestra.run.smithi103.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-04-24T14:29:32.279 INFO:teuthology.orchestra.run.smithi103.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2024-04-24T14:29:32.401 INFO:teuthology.orchestra.run.smithi173.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-915-g330067db-1focal [3,940 B] 2024-04-24T14:29:32.402 INFO:teuthology.orchestra.run.smithi173.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-915-g330067db-1focal [872 kB] 2024-04-24T14:29:32.415 INFO:teuthology.orchestra.run.smithi173.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-915-g330067db-1focal [2,415 kB] 2024-04-24T14:29:32.433 INFO:teuthology.orchestra.run.smithi173.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-915-g330067db-1focal [80.8 kB] 2024-04-24T14:29:32.433 INFO:teuthology.orchestra.run.smithi173.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-915-g330067db-1focal [119 kB] 2024-04-24T14:29:32.435 INFO:teuthology.orchestra.run.smithi173.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-915-g330067db-1focal [6,653 kB] 2024-04-24T14:29:32.659 INFO:teuthology.orchestra.run.smithi173.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-915-g330067db-1focal [7,890 kB] 2024-04-24T14:29:32.701 INFO:teuthology.orchestra.run.smithi103.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.2 [95.9 kB] 2024-04-24T14:29:32.731 INFO:teuthology.orchestra.run.smithi103.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-24T14:29:32.738 INFO:teuthology.orchestra.run.smithi103.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-04-24T14:29:32.787 INFO:teuthology.orchestra.run.smithi103.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-04-24T14:29:32.817 INFO:teuthology.orchestra.run.smithi103.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-04-24T14:29:32.842 INFO:teuthology.orchestra.run.smithi103.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-04-24T14:29:32.845 INFO:teuthology.orchestra.run.smithi103.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-04-24T14:29:32.853 INFO:teuthology.orchestra.run.smithi103.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-04-24T14:29:32.935 INFO:teuthology.orchestra.run.smithi173.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-915-g330067db-1focal [4,056 B] 2024-04-24T14:29:32.935 INFO:teuthology.orchestra.run.smithi173.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-915-g330067db-1focal [53.2 MB] 2024-04-24T14:29:33.340 INFO:teuthology.orchestra.run.smithi103.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-04-24T14:29:33.368 INFO:teuthology.orchestra.run.smithi103.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-04-24T14:29:33.534 INFO:teuthology.orchestra.run.smithi103.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-04-24T14:29:34.229 INFO:teuthology.orchestra.run.smithi103.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-04-24T14:29:34.230 INFO:teuthology.orchestra.run.smithi103.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-04-24T14:29:34.231 INFO:teuthology.orchestra.run.smithi103.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-04-24T14:29:34.232 INFO:teuthology.orchestra.run.smithi103.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-04-24T14:29:34.233 INFO:teuthology.orchestra.run.smithi103.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-04-24T14:29:34.235 INFO:teuthology.orchestra.run.smithi103.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-04-24T14:29:34.235 INFO:teuthology.orchestra.run.smithi103.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-04-24T14:29:34.238 INFO:teuthology.orchestra.run.smithi103.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-04-24T14:29:34.238 INFO:teuthology.orchestra.run.smithi103.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-04-24T14:29:34.310 INFO:teuthology.orchestra.run.smithi103.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-04-24T14:29:34.483 INFO:teuthology.orchestra.run.smithi103.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-04-24T14:29:34.866 INFO:teuthology.orchestra.run.smithi103.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-04-24T14:29:34.919 INFO:teuthology.orchestra.run.smithi103.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-04-24T14:29:34.954 INFO:teuthology.orchestra.run.smithi103.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-04-24T14:29:35.026 INFO:teuthology.orchestra.run.smithi103.stdout:Get:61 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-915-g330067db-1focal [3,558 kB] 2024-04-24T14:29:35.028 INFO:teuthology.orchestra.run.smithi090.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-915-g330067db-1focal [3,558 kB] 2024-04-24T14:29:35.054 INFO:teuthology.orchestra.run.smithi103.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-04-24T14:29:35.102 INFO:teuthology.orchestra.run.smithi103.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-04-24T14:29:35.165 INFO:teuthology.orchestra.run.smithi173.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-915-g330067db-1focal [229 kB] 2024-04-24T14:29:35.169 INFO:teuthology.orchestra.run.smithi173.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-915-g330067db-1focal [21.9 kB] 2024-04-24T14:29:35.169 INFO:teuthology.orchestra.run.smithi173.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-915-g330067db-1focal [17.1 MB] 2024-04-24T14:29:35.256 INFO:teuthology.orchestra.run.smithi103.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2024-04-24T14:29:35.266 INFO:teuthology.orchestra.run.smithi103.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-04-24T14:29:35.416 INFO:teuthology.orchestra.run.smithi103.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-04-24T14:29:35.417 INFO:teuthology.orchestra.run.smithi103.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-04-24T14:29:35.417 INFO:teuthology.orchestra.run.smithi103.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-04-24T14:29:35.582 INFO:teuthology.orchestra.run.smithi103.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-04-24T14:29:35.666 INFO:teuthology.orchestra.run.smithi103.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-04-24T14:29:35.762 INFO:teuthology.orchestra.run.smithi173.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-915-g330067db-1focal [80.6 kB] 2024-04-24T14:29:35.763 INFO:teuthology.orchestra.run.smithi173.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 rbd-nbd amd64 17.2.7-915-g330067db-1focal [166 kB] 2024-04-24T14:29:35.857 INFO:teuthology.orchestra.run.smithi103.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-04-24T14:29:36.191 INFO:teuthology.orchestra.run.smithi103.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-04-24T14:29:36.193 INFO:teuthology.orchestra.run.smithi103.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-24T14:29:36.197 INFO:teuthology.orchestra.run.smithi103.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-04-24T14:29:36.199 INFO:teuthology.orchestra.run.smithi103.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-04-24T14:29:36.209 INFO:teuthology.orchestra.run.smithi103.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-04-24T14:29:36.213 INFO:teuthology.orchestra.run.smithi103.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-04-24T14:29:36.215 INFO:teuthology.orchestra.run.smithi103.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-04-24T14:29:36.215 INFO:teuthology.orchestra.run.smithi103.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-04-24T14:29:36.390 INFO:teuthology.orchestra.run.smithi103.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-04-24T14:29:36.408 INFO:teuthology.orchestra.run.smithi173.stdout:Fetched 201 MB in 6s (31.9 MB/s) 2024-04-24T14:29:36.579 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-04-24T14:29:36.635 INFO:teuthology.orchestra.run.smithi173.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141729 files and directories currently installed.) 2024-04-24T14:29:36.640 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-04-24T14:29:36.645 INFO:teuthology.orchestra.run.smithi103.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-04-24T14:29:36.725 INFO:teuthology.orchestra.run.smithi090.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-915-g330067db-1focal [761 kB] 2024-04-24T14:29:36.744 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-04-24T14:29:36.972 INFO:teuthology.orchestra.run.smithi103.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-915-g330067db-1focal [761 kB] 2024-04-24T14:29:36.992 INFO:teuthology.orchestra.run.smithi090.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-915-g330067db-1focal [412 kB] 2024-04-24T14:29:37.057 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../001-librbd1_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:37.134 INFO:teuthology.orchestra.run.smithi090.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-915-g330067db-1focal [22.6 kB] 2024-04-24T14:29:37.135 INFO:teuthology.orchestra.run.smithi090.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-915-g330067db-1focal [200 kB] 2024-04-24T14:29:37.181 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking librbd1 (17.2.7-915-g330067db-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-04-24T14:29:37.205 INFO:teuthology.orchestra.run.smithi090.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-915-g330067db-1focal [47.8 kB] 2024-04-24T14:29:37.224 INFO:teuthology.orchestra.run.smithi090.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-915-g330067db-1focal [386 kB] 2024-04-24T14:29:37.276 INFO:teuthology.orchestra.run.smithi103.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-915-g330067db-1focal [412 kB] 2024-04-24T14:29:37.337 INFO:teuthology.orchestra.run.smithi090.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-915-g330067db-1focal [5,601 kB] 2024-04-24T14:29:37.426 INFO:teuthology.orchestra.run.smithi103.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-915-g330067db-1focal [22.6 kB] 2024-04-24T14:29:37.427 INFO:teuthology.orchestra.run.smithi103.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-915-g330067db-1focal [200 kB] 2024-04-24T14:29:37.498 INFO:teuthology.orchestra.run.smithi103.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-915-g330067db-1focal [47.8 kB] 2024-04-24T14:29:37.515 INFO:teuthology.orchestra.run.smithi103.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-915-g330067db-1focal [386 kB] 2024-04-24T14:29:37.657 INFO:teuthology.orchestra.run.smithi103.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-915-g330067db-1focal [5,601 kB] 2024-04-24T14:29:37.871 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../002-librados2_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:38.011 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking librados2 (17.2.7-915-g330067db-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-04-24T14:29:38.730 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libcephfs2. 2024-04-24T14:29:38.748 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:38.789 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libcephfs2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:38.908 INFO:teuthology.orchestra.run.smithi090.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-915-g330067db-1focal [116 kB] 2024-04-24T14:29:38.941 INFO:teuthology.orchestra.run.smithi090.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-915-g330067db-1focal [479 kB] 2024-04-24T14:29:39.056 INFO:teuthology.orchestra.run.smithi090.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-915-g330067db-1focal [26.9 MB] 2024-04-24T14:29:39.075 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-rados. 2024-04-24T14:29:39.095 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../004-python3-rados_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:39.125 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-rados (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:39.336 INFO:teuthology.orchestra.run.smithi103.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-915-g330067db-1focal [116 kB] 2024-04-24T14:29:39.370 INFO:teuthology.orchestra.run.smithi103.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-915-g330067db-1focal [479 kB] 2024-04-24T14:29:39.483 INFO:teuthology.orchestra.run.smithi103.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-915-g330067db-1focal [26.9 MB] 2024-04-24T14:29:39.511 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-24T14:29:39.528 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:29:39.569 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-ceph-argparse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:39.822 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-cephfs. 2024-04-24T14:29:39.839 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:39.888 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-cephfs (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:40.191 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-24T14:29:40.208 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:29:40.241 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-ceph-common (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:40.494 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-prettytable. 2024-04-24T14:29:40.512 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-04-24T14:29:40.544 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-04-24T14:29:40.804 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-rbd. 2024-04-24T14:29:40.823 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:40.863 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-rbd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:41.216 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-04-24T14:29:41.233 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-04-24T14:29:41.274 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-04-24T14:29:41.619 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-04-24T14:29:41.639 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-04-24T14:29:41.677 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-04-24T14:29:41.955 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-24T14:29:41.972 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-04-24T14:29:42.013 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-04-24T14:29:42.408 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-24T14:29:42.413 INFO:teuthology.orchestra.run.smithi090.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-915-g330067db-1focal [5,170 kB] 2024-04-24T14:29:42.424 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-04-24T14:29:42.466 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-04-24T14:29:42.751 INFO:teuthology.orchestra.run.smithi090.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-915-g330067db-1focal [223 kB] 2024-04-24T14:29:42.757 INFO:teuthology.orchestra.run.smithi090.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-915-g330067db-1focal [158 kB] 2024-04-24T14:29:42.777 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-24T14:29:42.778 INFO:teuthology.orchestra.run.smithi090.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-915-g330067db-1focal [1,594 kB] 2024-04-24T14:29:42.795 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-04-24T14:29:42.860 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-04-24T14:29:42.871 INFO:teuthology.orchestra.run.smithi090.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-915-g330067db-1focal [4,686 kB] 2024-04-24T14:29:43.184 INFO:teuthology.orchestra.run.smithi090.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-915-g330067db-1focal [23.1 MB] 2024-04-24T14:29:43.238 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-24T14:29:43.257 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-04-24T14:29:43.307 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:29:43.825 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-24T14:29:43.842 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-04-24T14:29:43.883 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:29:44.278 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-24T14:29:44.294 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-04-24T14:29:44.336 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:29:44.502 INFO:teuthology.orchestra.run.smithi090.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-915-g330067db-1focal [3,940 B] 2024-04-24T14:29:44.503 INFO:teuthology.orchestra.run.smithi090.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-915-g330067db-1focal [872 kB] 2024-04-24T14:29:44.517 INFO:teuthology.orchestra.run.smithi090.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-915-g330067db-1focal [2,415 kB] 2024-04-24T14:29:44.553 INFO:teuthology.orchestra.run.smithi090.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-915-g330067db-1focal [80.8 kB] 2024-04-24T14:29:44.568 INFO:teuthology.orchestra.run.smithi090.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-915-g330067db-1focal [119 kB] 2024-04-24T14:29:44.576 INFO:teuthology.orchestra.run.smithi090.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-915-g330067db-1focal [6,653 kB] 2024-04-24T14:29:44.681 INFO:teuthology.orchestra.run.smithi103.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-915-g330067db-1focal [5,170 kB] 2024-04-24T14:29:44.697 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-04-24T14:29:44.715 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-04-24T14:29:44.747 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-04-24T14:29:44.846 INFO:teuthology.orchestra.run.smithi090.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-915-g330067db-1focal [7,890 kB] 2024-04-24T14:29:45.126 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package librgw2. 2024-04-24T14:29:45.142 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../019-librgw2_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:45.179 INFO:teuthology.orchestra.run.smithi090.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-915-g330067db-1focal [4,056 B] 2024-04-24T14:29:45.180 INFO:teuthology.orchestra.run.smithi090.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-915-g330067db-1focal [53.2 MB] 2024-04-24T14:29:45.183 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking librgw2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:45.373 INFO:teuthology.orchestra.run.smithi103.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-915-g330067db-1focal [223 kB] 2024-04-24T14:29:45.414 INFO:teuthology.orchestra.run.smithi103.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-915-g330067db-1focal [158 kB] 2024-04-24T14:29:45.419 INFO:teuthology.orchestra.run.smithi103.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-915-g330067db-1focal [1,594 kB] 2024-04-24T14:29:45.653 INFO:teuthology.orchestra.run.smithi103.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-915-g330067db-1focal [4,686 kB] 2024-04-24T14:29:45.961 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-rgw. 2024-04-24T14:29:45.978 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:46.019 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-rgw (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:46.285 INFO:teuthology.orchestra.run.smithi103.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-915-g330067db-1focal [23.1 MB] 2024-04-24T14:29:46.388 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package liboath0. 2024-04-24T14:29:46.405 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-04-24T14:29:46.466 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-04-24T14:29:47.016 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libradosstriper1. 2024-04-24T14:29:47.032 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:47.074 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libradosstriper1 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:47.536 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-common. 2024-04-24T14:29:47.553 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../023-ceph-common_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:47.594 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-common (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:47.743 INFO:teuthology.orchestra.run.smithi090.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-915-g330067db-1focal [229 kB] 2024-04-24T14:29:47.748 INFO:teuthology.orchestra.run.smithi090.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-915-g330067db-1focal [21.9 kB] 2024-04-24T14:29:47.749 INFO:teuthology.orchestra.run.smithi090.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-915-g330067db-1focal [17.1 MB] 2024-04-24T14:29:48.433 INFO:teuthology.orchestra.run.smithi090.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-915-g330067db-1focal [80.6 kB] 2024-04-24T14:29:48.434 INFO:teuthology.orchestra.run.smithi090.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 rbd-nbd amd64 17.2.7-915-g330067db-1focal [166 kB] 2024-04-24T14:29:48.718 INFO:teuthology.orchestra.run.smithi103.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-915-g330067db-1focal [3,940 B] 2024-04-24T14:29:48.719 INFO:teuthology.orchestra.run.smithi103.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-915-g330067db-1focal [872 kB] 2024-04-24T14:29:48.732 INFO:teuthology.orchestra.run.smithi103.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-915-g330067db-1focal [2,415 kB] 2024-04-24T14:29:48.826 INFO:teuthology.orchestra.run.smithi103.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-915-g330067db-1focal [80.8 kB] 2024-04-24T14:29:48.828 INFO:teuthology.orchestra.run.smithi103.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-915-g330067db-1focal [119 kB] 2024-04-24T14:29:48.831 INFO:teuthology.orchestra.run.smithi103.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-915-g330067db-1focal [6,653 kB] 2024-04-24T14:29:49.069 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 201 MB in 19s (10.6 MB/s) 2024-04-24T14:29:49.129 INFO:teuthology.orchestra.run.smithi103.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-915-g330067db-1focal [7,890 kB] 2024-04-24T14:29:49.205 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-04-24T14:29:49.264 INFO:teuthology.orchestra.run.smithi090.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141729 files and directories currently installed.) 2024-04-24T14:29:49.268 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-04-24T14:29:49.354 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-04-24T14:29:49.432 INFO:teuthology.orchestra.run.smithi103.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-915-g330067db-1focal [4,056 B] 2024-04-24T14:29:49.432 INFO:teuthology.orchestra.run.smithi103.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-915-g330067db-1focal [53.2 MB] 2024-04-24T14:29:49.701 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../001-librbd1_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:49.825 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librbd1 (17.2.7-915-g330067db-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-04-24T14:29:50.175 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-base. 2024-04-24T14:29:50.192 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../024-ceph-base_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:50.243 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-base (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:50.532 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../002-librados2_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:50.656 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librados2 (17.2.7-915-g330067db-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-04-24T14:29:51.121 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-cryptography. 2024-04-24T14:29:51.140 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-04-24T14:29:51.179 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-04-24T14:29:51.386 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libcephfs2. 2024-04-24T14:29:51.404 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:51.445 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libcephfs2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:51.549 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-openssl. 2024-04-24T14:29:51.565 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-04-24T14:29:51.580 INFO:teuthology.orchestra.run.smithi103.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-915-g330067db-1focal [229 kB] 2024-04-24T14:29:51.585 INFO:teuthology.orchestra.run.smithi103.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-915-g330067db-1focal [21.9 kB] 2024-04-24T14:29:51.585 INFO:teuthology.orchestra.run.smithi103.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-915-g330067db-1focal [17.1 MB] 2024-04-24T14:29:51.607 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-04-24T14:29:51.772 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rados. 2024-04-24T14:29:51.789 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../004-python3-rados_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:51.831 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rados (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:51.910 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-24T14:29:51.926 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:29:51.968 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-modules-core (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:52.098 INFO:teuthology.orchestra.run.smithi103.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-915-g330067db-1focal [80.6 kB] 2024-04-24T14:29:52.144 INFO:teuthology.orchestra.run.smithi103.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/ubuntu/focal/flavors/default focal/main amd64 rbd-nbd amd64 17.2.7-915-g330067db-1focal [166 kB] 2024-04-24T14:29:52.192 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-24T14:29:52.208 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:29:52.250 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-ceph-argparse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:52.314 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-24T14:29:52.332 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-04-24T14:29:52.373 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-04-24T14:29:52.538 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cephfs. 2024-04-24T14:29:52.556 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:52.596 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cephfs (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:52.650 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-24T14:29:52.668 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2024-04-24T14:29:52.708 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-04-24T14:29:52.778 INFO:teuthology.orchestra.run.smithi103.stdout:Fetched 201 MB in 22s (8,924 kB/s) 2024-04-24T14:29:52.916 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-24T14:29:52.933 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:29:52.947 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-04-24T14:29:52.974 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-ceph-common (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:53.004 INFO:teuthology.orchestra.run.smithi103.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141729 files and directories currently installed.) 2024-04-24T14:29:53.008 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-04-24T14:29:53.028 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-natsort. 2024-04-24T14:29:53.048 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2024-04-24T14:29:53.078 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-04-24T14:29:53.095 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-04-24T14:29:53.278 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-prettytable. 2024-04-24T14:29:53.294 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-04-24T14:29:53.336 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-04-24T14:29:53.407 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-jwt. 2024-04-24T14:29:53.422 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-04-24T14:29:53.465 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-04-24T14:29:53.465 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../001-librbd1_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:53.590 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rbd. 2024-04-24T14:29:53.591 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking librbd1 (17.2.7-915-g330067db-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-04-24T14:29:53.606 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:53.648 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rbd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:53.760 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-logutils. 2024-04-24T14:29:53.777 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2024-04-24T14:29:53.851 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-04-24T14:29:53.976 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-04-24T14:29:53.992 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-04-24T14:29:54.026 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-04-24T14:29:54.154 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-24T14:29:54.169 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-04-24T14:29:54.212 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-04-24T14:29:54.313 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../002-librados2_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:54.379 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-04-24T14:29:54.399 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-04-24T14:29:54.438 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking librados2 (17.2.7-915-g330067db-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-04-24T14:29:54.438 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-04-24T14:29:54.548 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-mako. 2024-04-24T14:29:54.565 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-04-24T14:29:54.606 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-04-24T14:29:54.809 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-24T14:29:54.825 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-04-24T14:29:54.867 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-04-24T14:29:54.934 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-24T14:29:54.952 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-24T14:29:54.993 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-24T14:29:55.125 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libcephfs2. 2024-04-24T14:29:55.144 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:55.175 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libcephfs2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:55.220 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-24T14:29:55.237 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-04-24T14:29:55.278 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-04-24T14:29:55.312 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-24T14:29:55.329 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-04-24T14:29:55.370 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-04-24T14:29:55.520 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-rados. 2024-04-24T14:29:55.539 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../004-python3-rados_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:55.578 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-rados (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:55.643 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-24T14:29:55.658 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-04-24T14:29:55.682 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-webob. 2024-04-24T14:29:55.698 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2024-04-24T14:29:55.699 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-04-24T14:29:55.740 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2024-04-24T14:29:55.972 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-24T14:29:55.989 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:29:56.031 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-ceph-argparse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:56.076 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-soupsieve. 2024-04-24T14:29:56.092 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-04-24T14:29:56.103 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-24T14:29:56.119 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-04-24T14:29:56.134 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-04-24T14:29:56.163 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:29:56.267 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-cephfs. 2024-04-24T14:29:56.286 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:56.350 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-cephfs (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:56.454 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-bs4. 2024-04-24T14:29:56.470 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2024-04-24T14:29:56.537 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-04-24T14:29:56.670 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-24T14:29:56.682 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-24T14:29:56.686 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:29:56.698 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-04-24T14:29:56.740 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:29:56.762 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-ceph-common (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:56.890 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-waitress. 2024-04-24T14:29:56.908 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-04-24T14:29:56.968 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-04-24T14:29:57.073 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-prettytable. 2024-04-24T14:29:57.089 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-04-24T14:29:57.094 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-24T14:29:57.112 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-04-24T14:29:57.152 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-04-24T14:29:57.153 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:29:57.302 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-tempita. 2024-04-24T14:29:57.320 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2024-04-24T14:29:57.360 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-04-24T14:29:57.409 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-rbd. 2024-04-24T14:29:57.426 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:57.468 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-rbd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:57.523 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-04-24T14:29:57.543 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-04-24T14:29:57.612 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-04-24T14:29:57.662 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-paste. 2024-04-24T14:29:57.680 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-04-24T14:29:57.721 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-04-24T14:29:57.812 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-04-24T14:29:57.832 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-04-24T14:29:57.862 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-04-24T14:29:57.977 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package librgw2. 2024-04-24T14:29:57.996 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../019-librgw2_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:58.035 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librgw2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:58.057 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-24T14:29:58.076 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-04-24T14:29:58.115 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-04-24T14:29:58.240 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-04-24T14:29:58.257 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-04-24T14:29:58.299 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-04-24T14:29:58.418 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-24T14:29:58.434 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2024-04-24T14:29:58.476 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-04-24T14:29:58.669 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-24T14:29:58.684 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-04-24T14:29:58.727 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-04-24T14:29:58.796 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-webtest. 2024-04-24T14:29:58.813 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2024-04-24T14:29:58.854 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-04-24T14:29:58.863 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rgw. 2024-04-24T14:29:58.879 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:58.913 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rgw (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:29:59.063 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-24T14:29:59.083 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-04-24T14:29:59.113 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-04-24T14:29:59.174 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pecan. 2024-04-24T14:29:59.192 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-04-24T14:29:59.241 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package liboath0. 2024-04-24T14:29:59.260 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-04-24T14:29:59.265 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-04-24T14:29:59.299 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-04-24T14:29:59.483 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-24T14:29:59.499 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-04-24T14:29:59.541 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-04-24T14:29:59.603 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-24T14:29:59.620 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libradosstriper1. 2024-04-24T14:29:59.621 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2024-04-24T14:29:59.637 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:29:59.661 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2024-04-24T14:29:59.679 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libradosstriper1 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:00.023 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-24T14:30:00.037 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-24T14:30:00.039 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:00.056 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-04-24T14:30:00.081 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:00.099 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-common. 2024-04-24T14:30:00.123 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../023-ceph-common_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:00.131 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:30:00.157 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-common (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:00.434 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr. 2024-04-24T14:30:00.452 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../049-ceph-mgr_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:00.492 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:00.640 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-24T14:30:00.656 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-04-24T14:30:00.698 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:30:00.970 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mon. 2024-04-24T14:30:00.989 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../050-ceph-mon_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:01.029 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mon (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:01.060 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-24T14:30:01.077 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-04-24T14:30:01.118 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:30:01.858 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-osd. 2024-04-24T14:30:01.873 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../051-ceph-osd_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:01.941 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-osd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:02.364 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-04-24T14:30:02.381 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-04-24T14:30:02.414 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-04-24T14:30:02.785 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package librgw2. 2024-04-24T14:30:02.801 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../019-librgw2_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:02.842 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking librgw2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:03.019 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-base. 2024-04-24T14:30:03.039 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../024-ceph-base_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:03.645 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-rgw. 2024-04-24T14:30:03.664 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:03.692 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-base (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:03.703 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-rgw (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:04.073 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package liboath0. 2024-04-24T14:30:04.089 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-04-24T14:30:04.123 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-04-24T14:30:04.294 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph. 2024-04-24T14:30:04.312 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../052-ceph_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:04.352 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:04.451 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libradosstriper1. 2024-04-24T14:30:04.471 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:04.509 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libradosstriper1 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:04.611 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cryptography. 2024-04-24T14:30:04.631 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-04-24T14:30:04.671 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-fuse. 2024-04-24T14:30:04.678 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-04-24T14:30:04.687 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../053-ceph-fuse_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:04.754 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-fuse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:04.996 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-common. 2024-04-24T14:30:05.013 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../023-ceph-common_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:05.042 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-openssl. 2024-04-24T14:30:05.054 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-common (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:05.057 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-04-24T14:30:05.125 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-04-24T14:30:05.149 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mds. 2024-04-24T14:30:05.168 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../054-ceph-mds_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:05.207 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mds (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:05.386 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-24T14:30:05.403 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:05.435 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-modules-core (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:05.752 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package cephadm. 2024-04-24T14:30:05.769 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../055-cephadm_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:05.810 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking cephadm (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:05.851 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-24T14:30:05.867 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-04-24T14:30:05.909 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-04-24T14:30:06.147 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-jinja2. 2024-04-24T14:30:06.167 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.2_all.deb ... 2024-04-24T14:30:06.212 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.2) ... 2024-04-24T14:30:06.237 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-24T14:30:06.253 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2024-04-24T14:30:06.287 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-04-24T14:30:06.523 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-24T14:30:06.543 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:06.581 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-cephadm (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:06.617 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-natsort. 2024-04-24T14:30:06.633 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2024-04-24T14:30:06.675 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-04-24T14:30:06.968 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-24T14:30:06.978 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-jwt. 2024-04-24T14:30:06.984 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2024-04-24T14:30:06.996 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-04-24T14:30:07.026 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-24T14:30:07.037 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-04-24T14:30:07.245 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libjs-underscore. 2024-04-24T14:30:07.264 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-04-24T14:30:07.295 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-04-24T14:30:07.365 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-logutils. 2024-04-24T14:30:07.382 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2024-04-24T14:30:07.423 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-04-24T14:30:07.502 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-base. 2024-04-24T14:30:07.518 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../024-ceph-base_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:07.562 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-base (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:07.580 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-04-24T14:30:07.597 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-04-24T14:30:07.647 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-04-24T14:30:07.720 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-24T14:30:07.737 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-04-24T14:30:07.778 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-04-24T14:30:08.017 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-routes. 2024-04-24T14:30:08.033 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2024-04-24T14:30:08.064 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-mako. 2024-04-24T14:30:08.075 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-04-24T14:30:08.081 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-04-24T14:30:08.122 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-04-24T14:30:08.428 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-24T14:30:08.434 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-cryptography. 2024-04-24T14:30:08.444 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:08.451 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-04-24T14:30:08.469 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-24T14:30:08.485 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-24T14:30:08.486 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-dashboard (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:08.492 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-04-24T14:30:08.527 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-24T14:30:08.822 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-24T14:30:08.838 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-04-24T14:30:08.880 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-04-24T14:30:08.887 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-openssl. 2024-04-24T14:30:08.907 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-04-24T14:30:08.946 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-04-24T14:30:09.192 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-webob. 2024-04-24T14:30:09.209 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2024-04-24T14:30:09.257 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-24T14:30:09.273 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:09.275 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2024-04-24T14:30:09.315 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mgr-modules-core (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:09.596 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-soupsieve. 2024-04-24T14:30:09.613 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-04-24T14:30:09.654 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-04-24T14:30:09.720 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-24T14:30:09.736 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-04-24T14:30:09.779 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-04-24T14:30:09.800 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-decorator. 2024-04-24T14:30:09.815 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-04-24T14:30:09.858 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-04-24T14:30:09.933 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-bs4. 2024-04-24T14:30:09.952 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2024-04-24T14:30:09.990 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-04-24T14:30:10.115 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-24T14:30:10.131 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2024-04-24T14:30:10.144 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-04-24T14:30:10.161 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-04-24T14:30:10.173 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-04-24T14:30:10.202 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-04-24T14:30:10.369 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-waitress. 2024-04-24T14:30:10.386 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-04-24T14:30:10.432 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-04-24T14:30:10.529 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-scipy. 2024-04-24T14:30:10.535 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-natsort. 2024-04-24T14:30:10.548 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-04-24T14:30:10.551 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2024-04-24T14:30:10.587 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-04-24T14:30:10.593 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-04-24T14:30:10.732 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-tempita. 2024-04-24T14:30:10.752 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2024-04-24T14:30:10.790 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-04-24T14:30:10.939 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-jwt. 2024-04-24T14:30:10.959 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-04-24T14:30:10.997 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-04-24T14:30:11.110 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-paste. 2024-04-24T14:30:11.128 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-04-24T14:30:11.168 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-04-24T14:30:11.342 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-logutils. 2024-04-24T14:30:11.358 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2024-04-24T14:30:11.400 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-04-24T14:30:11.490 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-24T14:30:11.507 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-04-24T14:30:11.548 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-04-24T14:30:11.728 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-24T14:30:11.745 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-04-24T14:30:11.786 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-04-24T14:30:11.834 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-24T14:30:11.853 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2024-04-24T14:30:11.884 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-04-24T14:30:11.962 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-04-24T14:30:11.981 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-04-24T14:30:12.020 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-04-24T14:30:12.115 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-mako. 2024-04-24T14:30:12.134 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-04-24T14:30:12.173 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-04-24T14:30:12.188 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-webtest. 2024-04-24T14:30:12.205 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2024-04-24T14:30:12.246 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-04-24T14:30:12.527 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-24T14:30:12.543 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-24T14:30:12.575 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pecan. 2024-04-24T14:30:12.585 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-24T14:30:12.591 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-04-24T14:30:12.607 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-joblib. 2024-04-24T14:30:12.623 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2024-04-24T14:30:12.625 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-04-24T14:30:12.665 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-04-24T14:30:12.913 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-24T14:30:12.929 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-04-24T14:30:12.971 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-04-24T14:30:12.980 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-24T14:30:12.997 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2024-04-24T14:30:13.038 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2024-04-24T14:30:13.061 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-sklearn. 2024-04-24T14:30:13.078 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-04-24T14:30:13.118 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-04-24T14:30:13.299 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-webob. 2024-04-24T14:30:13.316 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2024-04-24T14:30:13.358 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2024-04-24T14:30:13.377 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-24T14:30:13.396 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:13.468 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:13.736 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-soupsieve. 2024-04-24T14:30:13.756 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-04-24T14:30:13.794 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-04-24T14:30:13.813 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr. 2024-04-24T14:30:13.827 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-24T14:30:13.833 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../049-ceph-mgr_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:13.845 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:13.871 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:13.885 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:14.106 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-bs4. 2024-04-24T14:30:14.125 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2024-04-24T14:30:14.164 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-04-24T14:30:14.360 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mon. 2024-04-24T14:30:14.375 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../050-ceph-mon_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:14.418 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mon (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:14.534 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-waitress. 2024-04-24T14:30:14.551 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-04-24T14:30:14.595 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-04-24T14:30:14.963 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-cachetools. 2024-04-24T14:30:14.970 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-tempita. 2024-04-24T14:30:14.979 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2024-04-24T14:30:14.986 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2024-04-24T14:30:15.029 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-04-24T14:30:15.039 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-04-24T14:30:15.205 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-osd. 2024-04-24T14:30:15.224 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../051-ceph-osd_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:15.263 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-osd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:15.332 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-paste. 2024-04-24T14:30:15.345 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-24T14:30:15.349 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-04-24T14:30:15.361 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-04-24T14:30:15.398 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-04-24T14:30:15.402 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-04-24T14:30:15.694 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-24T14:30:15.711 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-04-24T14:30:15.756 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-24T14:30:15.773 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-04-24T14:30:15.776 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-04-24T14:30:15.814 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-04-24T14:30:16.080 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-24T14:30:16.097 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2024-04-24T14:30:16.125 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-rsa. 2024-04-24T14:30:16.138 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-04-24T14:30:16.142 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2024-04-24T14:30:16.183 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-rsa (4.0-3) ... 2024-04-24T14:30:16.483 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-webtest. 2024-04-24T14:30:16.499 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2024-04-24T14:30:16.535 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-google-auth. 2024-04-24T14:30:16.541 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-04-24T14:30:16.553 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2024-04-24T14:30:16.593 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-04-24T14:30:16.886 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pecan. 2024-04-24T14:30:16.902 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-04-24T14:30:16.913 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-blinker. 2024-04-24T14:30:16.929 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-04-24T14:30:16.944 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-04-24T14:30:16.971 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-04-24T14:30:17.299 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-oauthlib. 2024-04-24T14:30:17.315 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-04-24T14:30:17.316 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-24T14:30:17.336 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2024-04-24T14:30:17.357 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-04-24T14:30:17.374 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2024-04-24T14:30:17.392 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph. 2024-04-24T14:30:17.412 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../052-ceph_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:17.460 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:17.686 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-24T14:30:17.702 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-04-24T14:30:17.730 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-24T14:30:17.745 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:17.761 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-04-24T14:30:17.787 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:17.787 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-fuse. 2024-04-24T14:30:17.803 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../053-ceph-fuse_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:17.846 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-fuse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:18.039 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-websocket. 2024-04-24T14:30:18.058 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-04-24T14:30:18.088 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-04-24T14:30:18.157 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mgr. 2024-04-24T14:30:18.173 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../049-ceph-mgr_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:18.215 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mgr (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:18.300 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mds. 2024-04-24T14:30:18.320 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../054-ceph-mds_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:18.366 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mds (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:18.719 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mon. 2024-04-24T14:30:18.737 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../050-ceph-mon_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:18.776 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mon (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:18.792 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-24T14:30:18.809 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-04-24T14:30:18.850 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-04-24T14:30:18.963 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package cephadm. 2024-04-24T14:30:18.978 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../055-cephadm_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:19.020 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking cephadm (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:19.371 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-24T14:30:19.391 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:19.429 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:19.514 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-osd. 2024-04-24T14:30:19.531 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../051-ceph-osd_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:19.597 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-osd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:19.765 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-jinja2. 2024-04-24T14:30:19.784 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.2_all.deb ... 2024-04-24T14:30:19.823 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.2) ... 2024-04-24T14:30:19.851 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-24T14:30:19.867 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2024-04-24T14:30:19.909 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-04-24T14:30:20.135 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-24T14:30:20.153 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:20.193 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-cephadm (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:20.262 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-24T14:30:20.279 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-04-24T14:30:20.320 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-04-24T14:30:20.548 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-24T14:30:20.564 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2024-04-24T14:30:20.606 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-24T14:30:20.623 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package jq. 2024-04-24T14:30:20.643 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-04-24T14:30:20.681 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-04-24T14:30:20.884 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libjs-underscore. 2024-04-24T14:30:20.900 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-04-24T14:30:20.934 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package socat. 2024-04-24T14:30:20.942 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-04-24T14:30:20.952 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2024-04-24T14:30:20.992 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking socat (1.7.3.3-2) ... 2024-04-24T14:30:21.212 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-04-24T14:30:21.227 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-04-24T14:30:21.270 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-04-24T14:30:21.312 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package xmlstarlet. 2024-04-24T14:30:21.330 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-04-24T14:30:21.370 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-04-24T14:30:21.582 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-routes. 2024-04-24T14:30:21.598 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2024-04-24T14:30:21.640 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-04-24T14:30:21.725 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-test. 2024-04-24T14:30:21.743 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../086-ceph-test_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:21.767 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph. 2024-04-24T14:30:21.783 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-test (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:21.787 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../052-ceph_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:21.825 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:21.985 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-24T14:30:22.073 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:22.073 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-dashboard (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:22.162 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-fuse. 2024-04-24T14:30:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../053-ceph-fuse_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:22.211 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-fuse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:22.665 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mds. 2024-04-24T14:30:22.681 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../054-ceph-mds_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:22.723 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mds (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:23.276 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package cephadm. 2024-04-24T14:30:23.293 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../055-cephadm_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:23.334 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking cephadm (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:23.559 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-decorator. 2024-04-24T14:30:23.577 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-04-24T14:30:23.617 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-04-24T14:30:23.688 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-jinja2. 2024-04-24T14:30:23.703 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.2_all.deb ... 2024-04-24T14:30:23.746 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.2) ... 2024-04-24T14:30:23.954 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-04-24T14:30:23.970 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-04-24T14:30:24.012 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-04-24T14:30:24.133 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-24T14:30:24.149 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:24.190 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mgr-cephadm (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:24.324 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-scipy. 2024-04-24T14:30:24.344 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-04-24T14:30:24.382 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-04-24T14:30:24.552 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-24T14:30:24.571 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2024-04-24T14:30:24.610 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-24T14:30:24.896 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libjs-underscore. 2024-04-24T14:30:24.916 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-04-24T14:30:24.954 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-04-24T14:30:25.249 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-04-24T14:30:25.265 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-04-24T14:30:25.307 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-04-24T14:30:25.652 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-routes. 2024-04-24T14:30:25.669 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2024-04-24T14:30:25.710 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-04-24T14:30:25.858 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-04-24T14:30:25.876 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-04-24T14:30:25.942 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-04-24T14:30:26.063 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-24T14:30:26.081 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:26.121 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mgr-dashboard (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:26.397 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-volume. 2024-04-24T14:30:26.414 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../087-ceph-volume_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:26.455 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-volume (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:26.486 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-joblib. 2024-04-24T14:30:26.503 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2024-04-24T14:30:26.544 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-04-24T14:30:26.835 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package fonts-mathjax. 2024-04-24T14:30:26.851 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-04-24T14:30:26.893 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-04-24T14:30:26.916 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-sklearn. 2024-04-24T14:30:26.934 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-04-24T14:30:27.008 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-04-24T14:30:27.413 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-24T14:30:27.430 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:27.471 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libcephfs-dev (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:27.544 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-decorator. 2024-04-24T14:30:27.564 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-04-24T14:30:27.769 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-24T14:30:27.789 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:27.835 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:27.874 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package nvme-cli. 2024-04-24T14:30:27.891 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2024-04-24T14:30:27.950 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2024-04-24T14:30:27.969 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-04-24T14:30:28.255 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-04-24T14:30:28.271 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-04-24T14:30:28.304 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libjs-mathjax. 2024-04-24T14:30:28.305 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-04-24T14:30:28.324 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-04-24T14:30:28.363 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-04-24T14:30:28.650 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-scipy. 2024-04-24T14:30:28.668 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-04-24T14:30:28.708 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-04-24T14:30:29.230 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cachetools. 2024-04-24T14:30:29.246 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2024-04-24T14:30:29.290 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-04-24T14:30:29.597 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-24T14:30:29.616 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-04-24T14:30:29.663 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-04-24T14:30:29.932 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python-natsort-doc. 2024-04-24T14:30:29.954 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2024-04-24T14:30:29.990 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-04-24T14:30:29.992 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-24T14:30:30.008 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-04-24T14:30:30.058 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-04-24T14:30:30.077 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-04-24T14:30:30.094 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-04-24T14:30:30.135 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-04-24T14:30:30.301 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-atomicwrites. 2024-04-24T14:30:30.318 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-04-24T14:30:30.350 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-04-24T14:30:30.388 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rsa. 2024-04-24T14:30:30.406 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2024-04-24T14:30:30.446 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rsa (4.0-3) ... 2024-04-24T14:30:30.648 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-attr. 2024-04-24T14:30:30.666 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2024-04-24T14:30:30.680 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-joblib. 2024-04-24T14:30:30.697 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2024-04-24T14:30:30.705 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-04-24T14:30:30.755 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-04-24T14:30:30.758 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-google-auth. 2024-04-24T14:30:30.775 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2024-04-24T14:30:30.816 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-04-24T14:30:31.067 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-webencodings. 2024-04-24T14:30:31.088 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-04-24T14:30:31.109 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-sklearn. 2024-04-24T14:30:31.121 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-blinker. 2024-04-24T14:30:31.124 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-04-24T14:30:31.128 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-04-24T14:30:31.138 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-04-24T14:30:31.184 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-04-24T14:30:31.203 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-04-24T14:30:31.428 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-html5lib. 2024-04-24T14:30:31.446 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2024-04-24T14:30:31.510 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-04-24T14:30:31.516 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-oauthlib. 2024-04-24T14:30:31.532 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-04-24T14:30:31.582 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-04-24T14:30:31.848 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-04-24T14:30:31.864 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-04-24T14:30:31.905 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-04-24T14:30:31.910 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-24T14:30:31.912 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-24T14:30:31.926 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:31.928 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-04-24T14:30:31.970 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-04-24T14:30:31.993 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:32.299 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-websocket. 2024-04-24T14:30:32.317 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-04-24T14:30:32.357 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-04-24T14:30:32.375 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-packaging. 2024-04-24T14:30:32.394 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2024-04-24T14:30:32.433 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-packaging (20.3-1) ... 2024-04-24T14:30:32.669 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-24T14:30:32.686 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-04-24T14:30:32.719 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pastescript. 2024-04-24T14:30:32.727 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-04-24T14:30:32.740 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-04-24T14:30:32.777 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-24T14:30:33.122 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pluggy. 2024-04-24T14:30:33.139 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-04-24T14:30:33.176 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-cachetools. 2024-04-24T14:30:33.180 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-04-24T14:30:33.195 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2024-04-24T14:30:33.233 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-04-24T14:30:33.434 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-24T14:30:33.453 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:33.500 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-psutil. 2024-04-24T14:30:33.500 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:33.520 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-04-24T14:30:33.570 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-24T14:30:33.583 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-04-24T14:30:33.586 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-04-24T14:30:33.628 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-04-24T14:30:33.897 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-24T14:30:33.911 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-py. 2024-04-24T14:30:33.916 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2024-04-24T14:30:33.932 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-04-24T14:30:33.955 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-04-24T14:30:33.969 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-04-24T14:30:34.006 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-24T14:30:34.022 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-04-24T14:30:34.064 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-04-24T14:30:34.323 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-24T14:30:34.340 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-04-24T14:30:34.352 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-24T14:30:34.368 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-04-24T14:30:34.381 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-04-24T14:30:34.400 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-rsa. 2024-04-24T14:30:34.410 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-04-24T14:30:34.420 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2024-04-24T14:30:34.459 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-rsa (4.0-3) ... 2024-04-24T14:30:34.688 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package jq. 2024-04-24T14:30:34.694 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-24T14:30:34.704 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-04-24T14:30:34.712 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-04-24T14:30:34.753 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-04-24T14:30:34.754 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-04-24T14:30:34.795 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-google-auth. 2024-04-24T14:30:34.811 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2024-04-24T14:30:34.853 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-04-24T14:30:35.008 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package socat. 2024-04-24T14:30:35.024 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2024-04-24T14:30:35.047 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pytest. 2024-04-24T14:30:35.067 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking socat (1.7.3.3-2) ... 2024-04-24T14:30:35.068 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-04-24T14:30:35.105 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-04-24T14:30:35.199 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-blinker. 2024-04-24T14:30:35.214 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-04-24T14:30:35.256 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-04-24T14:30:35.421 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package xmlstarlet. 2024-04-24T14:30:35.438 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-04-24T14:30:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-simplejson. 2024-04-24T14:30:35.459 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-04-24T14:30:35.479 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-04-24T14:30:35.501 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-04-24T14:30:35.594 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-oauthlib. 2024-04-24T14:30:35.610 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-04-24T14:30:35.652 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-04-24T14:30:35.819 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-test. 2024-04-24T14:30:35.823 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-24T14:30:35.837 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../086-ceph-test_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:35.844 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-04-24T14:30:35.877 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-test (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:35.880 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-04-24T14:30:35.998 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-24T14:30:36.015 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-04-24T14:30:36.056 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-04-24T14:30:36.392 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-websocket. 2024-04-24T14:30:36.412 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-04-24T14:30:36.450 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-04-24T14:30:36.668 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package radosgw. 2024-04-24T14:30:36.688 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../108-radosgw_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:36.726 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking radosgw (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:36.779 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-24T14:30:36.797 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-04-24T14:30:36.837 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-04-24T14:30:37.409 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-24T14:30:37.425 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:37.467 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:37.939 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-24T14:30:37.959 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2024-04-24T14:30:37.997 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-04-24T14:30:38.384 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-24T14:30:38.401 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-04-24T14:30:38.432 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package rbd-fuse. 2024-04-24T14:30:38.442 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-04-24T14:30:38.452 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:38.490 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking rbd-fuse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:38.720 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package jq. 2024-04-24T14:30:38.736 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-04-24T14:30:38.759 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package rbd-nbd. 2024-04-24T14:30:38.775 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../110-rbd-nbd_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:38.795 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-04-24T14:30:38.817 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking rbd-nbd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:39.040 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package socat. 2024-04-24T14:30:39.059 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2024-04-24T14:30:39.098 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking socat (1.7.3.3-2) ... 2024-04-24T14:30:39.173 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-attr (19.3.0-2) ... 2024-04-24T14:30:39.444 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package xmlstarlet. 2024-04-24T14:30:39.461 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-04-24T14:30:39.474 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-04-24T14:30:39.502 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-04-24T14:30:39.600 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-04-24T14:30:39.748 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2024-04-24T14:30:39.842 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-test. 2024-04-24T14:30:39.861 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../086-ceph-test_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:39.900 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-test (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:39.937 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up cephadm (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:40.450 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-volume. 2024-04-24T14:30:40.467 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../087-ceph-volume_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:40.508 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-volume (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:40.515 INFO:teuthology.orchestra.run.smithi173.stdout:Adding system user cephadm....done 2024-04-24T14:30:40.677 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-04-24T14:30:40.803 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-04-24T14:30:40.865 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package fonts-mathjax. 2024-04-24T14:30:40.881 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-04-24T14:30:40.923 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-04-24T14:30:41.155 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-24T14:30:41.436 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-24T14:30:41.448 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-04-24T14:30:41.456 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:41.502 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libcephfs-dev (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:41.549 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-04-24T14:30:41.658 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-04-24T14:30:41.915 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package nvme-cli. 2024-04-24T14:30:41.933 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2024-04-24T14:30:41.991 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2024-04-24T14:30:41.993 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-04-24T14:30:42.320 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libjs-mathjax. 2024-04-24T14:30:42.340 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-04-24T14:30:42.379 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-04-24T14:30:42.388 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-04-24T14:30:42.664 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-04-24T14:30:42.957 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-04-24T14:30:43.241 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-04-24T14:30:43.526 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-ceph-argparse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:43.827 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-04-24T14:30:43.936 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-04-24T14:30:44.196 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-04-24T14:30:44.333 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python-natsort-doc. 2024-04-24T14:30:44.352 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2024-04-24T14:30:44.392 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-04-24T14:30:44.480 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-volume. 2024-04-24T14:30:44.496 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../087-ceph-volume_17.2.7-915-g330067db-1focal_all.deb ... 2024-04-24T14:30:44.497 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-04-24T14:30:44.537 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-volume (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:44.624 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-04-24T14:30:44.707 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-atomicwrites. 2024-04-24T14:30:44.726 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-04-24T14:30:44.765 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-04-24T14:30:44.878 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package fonts-mathjax. 2024-04-24T14:30:44.897 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-04-24T14:30:44.936 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-04-24T14:30:44.992 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-04-24T14:30:45.069 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-attr. 2024-04-24T14:30:45.089 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2024-04-24T14:30:45.126 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-04-24T14:30:45.293 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-04-24T14:30:45.439 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-webencodings. 2024-04-24T14:30:45.456 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-04-24T14:30:45.465 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-24T14:30:45.482 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:45.497 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-04-24T14:30:45.523 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libcephfs-dev (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:45.569 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-04-24T14:30:45.792 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-html5lib. 2024-04-24T14:30:45.810 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2024-04-24T14:30:45.858 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-04-24T14:30:45.912 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-04-24T14:30:45.943 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package nvme-cli. 2024-04-24T14:30:45.959 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2024-04-24T14:30:46.044 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2024-04-24T14:30:46.180 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.2) ... 2024-04-24T14:30:46.212 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-04-24T14:30:46.233 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-04-24T14:30:46.270 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-04-24T14:30:46.365 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libjs-mathjax. 2024-04-24T14:30:46.382 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-04-24T14:30:46.439 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-04-24T14:30:46.523 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-04-24T14:30:46.700 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-packaging. 2024-04-24T14:30:46.718 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2024-04-24T14:30:46.758 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-packaging (20.3-1) ... 2024-04-24T14:30:46.816 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-packaging (20.3-1) ... 2024-04-24T14:30:47.095 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pastescript. 2024-04-24T14:30:47.109 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-04-24T14:30:47.112 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-04-24T14:30:47.153 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-24T14:30:47.260 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-04-24T14:30:47.378 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-04-24T14:30:47.449 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pluggy. 2024-04-24T14:30:47.469 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-04-24T14:30:47.508 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-04-24T14:30:47.654 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-04-24T14:30:47.819 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-psutil. 2024-04-24T14:30:47.840 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-04-24T14:30:47.911 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-04-24T14:30:48.023 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2024-04-24T14:30:48.221 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python-natsort-doc. 2024-04-24T14:30:48.242 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2024-04-24T14:30:48.256 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-py. 2024-04-24T14:30:48.273 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-04-24T14:30:48.279 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-04-24T14:30:48.315 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-04-24T14:30:48.374 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-04-24T14:30:48.653 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-24T14:30:48.671 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-04-24T14:30:48.721 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-04-24T14:30:48.724 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-atomicwrites. 2024-04-24T14:30:48.741 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-04-24T14:30:48.743 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up socat (1.7.3.3-2) ... 2024-04-24T14:30:48.782 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-04-24T14:30:48.869 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-ceph-common (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:49.051 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-24T14:30:49.067 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-04-24T14:30:49.110 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-04-24T14:30:49.110 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-attr. 2024-04-24T14:30:49.127 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2024-04-24T14:30:49.162 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:30:49.168 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-04-24T14:30:49.405 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pytest. 2024-04-24T14:30:49.425 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-04-24T14:30:49.471 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-04-24T14:30:49.504 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-webencodings. 2024-04-24T14:30:49.524 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-04-24T14:30:49.562 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-04-24T14:30:49.796 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-simplejson. 2024-04-24T14:30:49.805 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-04-24T14:30:49.812 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-04-24T14:30:49.855 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-04-24T14:30:49.890 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-html5lib. 2024-04-24T14:30:49.907 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2024-04-24T14:30:49.948 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-04-24T14:30:50.165 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-04-24T14:30:50.291 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-04-24T14:30:50.310 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-04-24T14:30:50.327 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-04-24T14:30:50.368 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-04-24T14:30:50.417 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-04-24T14:30:50.692 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-24T14:30:50.710 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-04-24T14:30:50.713 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-04-24T14:30:50.758 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-04-24T14:30:50.854 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-packaging. 2024-04-24T14:30:50.871 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2024-04-24T14:30:50.912 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-packaging (20.3-1) ... 2024-04-24T14:30:51.012 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:30:51.113 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-04-24T14:30:51.214 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-04-24T14:30:51.232 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pastescript. 2024-04-24T14:30:51.253 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-04-24T14:30:51.290 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-24T14:30:51.506 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package radosgw. 2024-04-24T14:30:51.527 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../108-radosgw_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:51.533 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-04-24T14:30:51.572 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking radosgw (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:51.643 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pluggy. 2024-04-24T14:30:51.663 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-04-24T14:30:51.701 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-04-24T14:30:51.817 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-04-24T14:30:52.038 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-psutil. 2024-04-24T14:30:52.055 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-04-24T14:30:52.096 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-04-24T14:30:52.102 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-04-24T14:30:52.228 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-24T14:30:52.450 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-py. 2024-04-24T14:30:52.467 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-04-24T14:30:52.487 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-04-24T14:30:52.508 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-04-24T14:30:52.775 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-04-24T14:30:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-24T14:30:52.889 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-04-24T14:30:52.922 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-04-24T14:30:53.077 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-04-24T14:30:53.258 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-24T14:30:53.275 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-04-24T14:30:53.317 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-04-24T14:30:53.420 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package rbd-fuse. 2024-04-24T14:30:53.441 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:53.460 INFO:teuthology.orchestra.run.smithi173.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-04-24T14:30:53.478 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking rbd-fuse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:53.652 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pytest. 2024-04-24T14:30:53.670 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-04-24T14:30:53.710 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-04-24T14:30:53.721 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-04-24T14:30:53.757 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package rbd-nbd. 2024-04-24T14:30:53.774 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../110-rbd-nbd_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:53.815 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking rbd-nbd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:53.831 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-04-24T14:30:54.092 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-simplejson. 2024-04-24T14:30:54.112 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-04-24T14:30:54.144 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-attr (19.3.0-2) ... 2024-04-24T14:30:54.150 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-04-24T14:30:54.166 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-04-24T14:30:54.453 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-24T14:30:54.463 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-04-24T14:30:54.470 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-04-24T14:30:54.511 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-04-24T14:30:54.517 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-04-24T14:30:54.590 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-04-24T14:30:54.643 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-04-24T14:30:54.716 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2024-04-24T14:30:54.794 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-webob (1:1.8.5-2) ... 2024-04-24T14:30:54.885 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up cephadm (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:55.121 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-04-24T14:30:55.318 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package radosgw. 2024-04-24T14:30:55.338 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../108-radosgw_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:55.375 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking radosgw (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:55.389 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up librados2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:55.431 INFO:teuthology.orchestra.run.smithi090.stdout:Adding system user cephadm....done 2024-04-24T14:30:55.507 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-04-24T14:30:55.585 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-04-24T14:30:55.711 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-04-24T14:30:56.055 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-24T14:30:56.348 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-04-24T14:30:56.457 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-04-24T14:30:56.584 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-04-24T14:30:56.911 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-04-24T14:30:56.966 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libsqlite3-mod-ceph (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:57.092 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-04-24T14:30:57.145 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package rbd-fuse. 2024-04-24T14:30:57.163 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:57.203 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking rbd-fuse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:57.218 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-04-24T14:30:57.314 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-04-24T14:30:57.482 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package rbd-nbd. 2024-04-24T14:30:57.499 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../110-rbd-nbd_17.2.7-915-g330067db-1focal_amd64.deb ... 2024-04-24T14:30:57.531 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking rbd-nbd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:57.586 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-04-24T14:30:57.598 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-04-24T14:30:57.871 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-04-24T14:30:57.874 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-attr (19.3.0-2) ... 2024-04-24T14:30:57.883 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-04-24T14:30:58.147 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:30:58.169 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-04-24T14:30:58.169 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-04-24T14:30:58.275 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-04-24T14:30:58.312 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-04-24T14:30:58.413 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2024-04-24T14:30:58.462 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-ceph-argparse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:58.610 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libcephfs2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:58.752 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-04-24T14:30:58.755 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-04-24T14:30:58.856 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-04-24T14:30:58.923 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up cephadm (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:59.079 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libradosstriper1 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:30:59.116 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-04-24T14:30:59.181 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-04-24T14:30:59.425 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-04-24T14:30:59.507 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-04-24T14:30:59.510 INFO:teuthology.orchestra.run.smithi103.stdout:Adding system user cephadm....done 2024-04-24T14:30:59.553 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-04-24T14:30:59.696 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-04-24T14:30:59.797 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-04-24T14:30:59.921 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-04-24T14:31:00.133 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-24T14:31:00.214 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-04-24T14:31:00.434 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-04-24T14:31:00.482 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-04-24T14:31:00.538 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up librbd1 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:00.593 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-04-24T14:31:00.639 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-04-24T14:31:00.719 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-04-24T14:31:00.774 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-04-24T14:31:00.817 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-04-24T14:31:00.900 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-04-24T14:31:01.026 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-rsa (4.0-3) ... 2024-04-24T14:31:01.037 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-04-24T14:31:01.110 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.2) ... 2024-04-24T14:31:01.319 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-04-24T14:31:01.429 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-04-24T14:31:01.431 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-04-24T14:31:01.470 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-04-24T14:31:01.699 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-04-24T14:31:01.739 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-packaging (20.3-1) ... 2024-04-24T14:31:01.763 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-modules-core (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:01.864 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-fuse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:02.017 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-04-24T14:31:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-04-24T14:31:02.083 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T14:31:02.083 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T14:31:02.158 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-04-24T14:31:02.284 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-04-24T14:31:02.301 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-04-24T14:31:02.524 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libcephfs-dev (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:02.569 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-04-24T14:31:02.619 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-ceph-argparse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:02.736 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-04-24T14:31:02.912 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-04-24T14:31:02.947 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2024-04-24T14:31:03.062 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-24T14:31:03.071 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-04-24T14:31:03.315 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-04-24T14:31:03.330 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-04-24T14:31:03.355 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-04-24T14:31:03.598 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-04-24T14:31:03.676 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up socat (1.7.3.3-2) ... 2024-04-24T14:31:03.700 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-rados (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:03.725 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-04-24T14:31:03.794 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-ceph-common (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:03.826 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up librgw2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:03.952 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-rbd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:04.078 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up rbd-fuse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:04.096 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:31:04.109 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-04-24T14:31:04.204 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up rbd-nbd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:04.230 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-04-24T14:31:04.347 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-routes (2.4.1-2) ... 2024-04-24T14:31:04.402 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-04-24T14:31:04.616 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-04-24T14:31:04.641 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-04-24T14:31:04.703 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-04-24T14:31:04.742 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-04-24T14:31:04.868 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-04-24T14:31:04.942 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-rgw (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:05.060 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-cephfs (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:05.062 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-04-24T14:31:05.137 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-04-24T14:31:05.186 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-04-24T14:31:05.389 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.2) ... 2024-04-24T14:31:05.430 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:31:05.556 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-04-24T14:31:05.708 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-04-24T14:31:05.732 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-04-24T14:31:06.019 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-04-24T14:31:06.033 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-packaging (20.3-1) ... 2024-04-24T14:31:06.115 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-common (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:06.287 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-04-24T14:31:06.359 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-04-24T14:31:06.477 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-04-24T14:31:06.563 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-04-24T14:31:06.577 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-04-24T14:31:06.635 INFO:teuthology.orchestra.run.smithi173.stdout:Adding group ceph....done 2024-04-24T14:31:06.714 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-24T14:31:06.854 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-04-24T14:31:06.992 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-04-24T14:31:07.037 INFO:teuthology.orchestra.run.smithi173.stdout:Adding system user ceph....done 2024-04-24T14:31:07.196 INFO:teuthology.orchestra.run.smithi173.stdout:Setting system user ceph properties....done 2024-04-24T14:31:07.218 INFO:teuthology.orchestra.run.smithi173.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-24T14:31:07.263 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-04-24T14:31:07.264 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2024-04-24T14:31:07.366 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-24T14:31:07.548 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-04-24T14:31:07.632 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-04-24T14:31:07.688 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-24T14:31:07.906 INFO:teuthology.orchestra.run.smithi090.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-04-24T14:31:08.034 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up socat (1.7.3.3-2) ... 2024-04-24T14:31:08.134 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-test (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:08.135 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-04-24T14:31:08.160 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-ceph-common (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:08.253 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-04-24T14:31:08.268 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up radosgw (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:08.478 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:31:08.530 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T14:31:08.530 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T14:31:08.563 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-04-24T14:31:08.612 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-04-24T14:31:08.906 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-04-24T14:31:08.955 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-04-24T14:31:09.033 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-04-24T14:31:09.081 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-04-24T14:31:09.159 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-webob (1:1.8.5-2) ... 2024-04-24T14:31:09.195 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-base (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:09.207 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-04-24T14:31:09.495 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-04-24T14:31:09.533 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-04-24T14:31:09.580 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-24T14:31:09.763 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librados2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:09.843 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:31:09.905 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-04-24T14:31:09.969 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-04-24T14:31:10.023 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mds (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:10.120 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-04-24T14:31:10.250 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T14:31:10.250 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T14:31:10.430 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-04-24T14:31:10.683 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:10.748 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-04-24T14:31:10.914 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T14:31:10.914 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T14:31:11.040 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-04-24T14:31:11.169 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-24T14:31:11.360 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-osd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:11.366 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libsqlite3-mod-ceph (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:11.437 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-04-24T14:31:11.492 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-04-24T14:31:11.618 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-04-24T14:31:11.755 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-04-24T14:31:11.840 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T14:31:11.841 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T14:31:12.005 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-04-24T14:31:12.039 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-04-24T14:31:12.281 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-04-24T14:31:12.295 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-k8sevents (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:12.441 INFO:teuthology.orchestra.run.smithi103.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-04-24T14:31:12.446 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:12.566 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:31:12.589 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mon (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:12.694 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-04-24T14:31:12.708 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-04-24T14:31:12.814 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T14:31:12.814 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T14:31:12.834 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-04-24T14:31:13.029 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libcephfs2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:13.152 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-04-24T14:31:13.155 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-04-24T14:31:13.240 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-cephadm (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:13.366 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:13.475 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libradosstriper1 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:13.492 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-dashboard (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:13.512 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-04-24T14:31:13.601 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-04-24T14:31:13.627 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-volume (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:13.638 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-04-24T14:31:13.764 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-webob (1:1.8.5-2) ... 2024-04-24T14:31:13.796 INFO:teuthology.orchestra.run.smithi173.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-04-24T14:31:13.947 INFO:teuthology.orchestra.run.smithi173.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-04-24T14:31:13.954 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-04-24T14:31:14.124 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-04-24T14:31:14.224 INFO:teuthology.orchestra.run.smithi173.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-04-24T14:31:14.450 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up librados2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:14.612 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-04-24T14:31:15.027 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librbd1 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:15.153 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-04-24T14:31:15.279 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-04-24T14:31:15.405 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-04-24T14:31:15.506 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rsa (4.0-3) ... 2024-04-24T14:31:15.801 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-04-24T14:31:15.927 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-04-24T14:31:16.137 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libsqlite3-mod-ceph (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:16.253 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-modules-core (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:16.263 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-04-24T14:31:16.380 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-fuse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:16.414 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-04-24T14:31:16.605 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T14:31:16.605 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T14:31:16.833 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-04-24T14:31:17.039 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libcephfs-dev (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:17.135 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-04-24T14:31:17.161 INFO:teuthology.orchestra.run.smithi173.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-04-24T14:31:17.277 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-04-24T14:31:17.486 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-04-24T14:31:17.563 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-24T14:31:17.632 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-04-24T14:31:17.760 INFO:teuthology.orchestra.run.smithi173.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-24T14:31:17.767 DEBUG:teuthology.parallel:result is None 2024-04-24T14:31:17.873 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-04-24T14:31:17.964 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libcephfs2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:18.115 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-04-24T14:31:18.200 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rados (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:18.328 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librgw2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:18.454 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rbd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:18.463 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libradosstriper1 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:18.580 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up rbd-fuse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:18.609 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-04-24T14:31:18.706 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up rbd-nbd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:18.816 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-routes (2.4.1-2) ... 2024-04-24T14:31:18.936 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-04-24T14:31:19.117 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-04-24T14:31:19.437 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rgw (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:19.563 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cephfs (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:19.689 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-04-24T14:31:19.983 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up librbd1 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:20.142 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-04-24T14:31:20.268 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-04-24T14:31:20.394 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-04-24T14:31:20.546 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-rsa (4.0-3) ... 2024-04-24T14:31:20.585 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-common (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:20.855 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-04-24T14:31:21.006 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-04-24T14:31:21.131 INFO:teuthology.orchestra.run.smithi090.stdout:Adding group ceph....done 2024-04-24T14:31:21.307 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mgr-modules-core (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:21.466 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-fuse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:21.693 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T14:31:21.693 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T14:31:21.950 INFO:teuthology.orchestra.run.smithi090.stdout:Adding system user ceph....done 2024-04-24T14:31:22.128 INFO:teuthology.orchestra.run.smithi090.stdout:Setting system user ceph properties....done 2024-04-24T14:31:22.143 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libcephfs-dev (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:22.146 INFO:teuthology.orchestra.run.smithi090.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-24T14:31:22.299 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-24T14:31:22.396 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-04-24T14:31:22.610 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-24T14:31:22.664 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-24T14:31:22.957 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-04-24T14:31:23.083 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-test (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:23.208 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up radosgw (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:23.343 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-rados (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:23.497 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up librgw2 (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:23.744 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T14:31:23.744 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T14:31:23.744 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-rbd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:23.746 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up rbd-fuse (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:23.872 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up rbd-nbd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:23.982 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-routes (2.4.1-2) ... 2024-04-24T14:31:24.128 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-base (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:24.258 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-04-24T14:31:24.508 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-24T14:31:24.576 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-rgw (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:24.727 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-cephfs (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:24.854 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-04-24T14:31:24.971 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mds (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:25.220 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T14:31:25.220 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T14:31:25.692 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:25.757 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-common (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:25.919 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T14:31:25.920 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T14:31:26.326 INFO:teuthology.orchestra.run.smithi103.stdout:Adding group ceph....done 2024-04-24T14:31:26.536 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-osd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:26.729 INFO:teuthology.orchestra.run.smithi103.stdout:Adding system user ceph....done 2024-04-24T14:31:26.930 INFO:teuthology.orchestra.run.smithi103.stdout:Setting system user ceph properties....done 2024-04-24T14:31:26.956 INFO:teuthology.orchestra.run.smithi103.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-24T14:31:27.032 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T14:31:27.033 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T14:31:27.102 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-24T14:31:27.425 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-24T14:31:27.496 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-k8sevents (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:27.622 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:27.767 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mon (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:27.884 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-test (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:27.993 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T14:31:27.993 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T14:31:28.010 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up radosgw (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:28.316 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T14:31:28.317 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T14:31:28.435 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-cephadm (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:28.604 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:28.729 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-dashboard (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:28.872 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-volume (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:28.962 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-base (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:29.000 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-04-24T14:31:29.152 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-04-24T14:31:29.330 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-24T14:31:29.445 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-04-24T14:31:30.015 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mds (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:30.253 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T14:31:30.254 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T14:31:30.691 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mgr (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:30.917 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T14:31:30.917 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T14:31:31.360 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-osd (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:31.902 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T14:31:31.902 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T14:31:32.344 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mgr-k8sevents (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:32.470 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:32.613 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-04-24T14:31:32.613 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mon (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:32.866 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T14:31:32.866 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T14:31:33.253 INFO:teuthology.orchestra.run.smithi090.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-24T14:31:33.258 DEBUG:teuthology.parallel:result is None 2024-04-24T14:31:33.323 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mgr-cephadm (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:33.465 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:33.625 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mgr-dashboard (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:33.768 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-volume (17.2.7-915-g330067db-1focal) ... 2024-04-24T14:31:33.895 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-04-24T14:31:34.071 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-04-24T14:31:34.398 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-04-24T14:31:37.283 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-04-24T14:31:37.861 INFO:teuthology.orchestra.run.smithi103.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-24T14:31:37.865 DEBUG:teuthology.parallel:result is None 2024-04-24T14:31:37.866 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T14:31:38.093 DEBUG:teuthology.orchestra.run.smithi090:> dpkg-query -W -f '${Version}' ceph 2024-04-24T14:31:38.116 INFO:teuthology.orchestra.run.smithi090.stdout:17.2.7-915-g330067db-1focal 2024-04-24T14:31:38.117 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-915-g330067db-1focal 2024-04-24T14:31:38.117 INFO:teuthology.task.install:The correct ceph version 17.2.7-915-g330067db-1focal is installed. 2024-04-24T14:31:38.119 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T14:31:38.336 DEBUG:teuthology.orchestra.run.smithi103:> dpkg-query -W -f '${Version}' ceph 2024-04-24T14:31:38.360 INFO:teuthology.orchestra.run.smithi103.stdout:17.2.7-915-g330067db-1focal 2024-04-24T14:31:38.360 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-915-g330067db-1focal 2024-04-24T14:31:38.360 INFO:teuthology.task.install:The correct ceph version 17.2.7-915-g330067db-1focal is installed. 2024-04-24T14:31:38.362 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T14:31:38.582 DEBUG:teuthology.orchestra.run.smithi173:> dpkg-query -W -f '${Version}' ceph 2024-04-24T14:31:38.606 INFO:teuthology.orchestra.run.smithi173.stdout:17.2.7-915-g330067db-1focal 2024-04-24T14:31:38.607 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-915-g330067db-1focal 2024-04-24T14:31:38.607 INFO:teuthology.task.install:The correct ceph version 17.2.7-915-g330067db-1focal is installed. 2024-04-24T14:31:38.608 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-24T14:31:38.609 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:31:38.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T14:31:38.626 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:31:38.626 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T14:31:38.644 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-24T14:31:38.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T14:31:38.664 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-24T14:31:38.664 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:31:38.664 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T14:31:38.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T14:31:38.696 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:31:38.696 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T14:31:38.712 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T14:31:38.770 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-24T14:31:38.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T14:31:38.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T14:31:38.846 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-24T14:31:38.846 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:31:38.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T14:31:38.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T14:31:38.921 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:31:38.921 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T14:31:38.937 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T14:31:38.948 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-24T14:31:38.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T14:31:38.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T14:31:38.976 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-24T14:31:38.976 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:31:38.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T14:31:38.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T14:31:39.050 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:31:39.050 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T14:31:39.066 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T14:31:39.076 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-24T14:31:39.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T14:31:39.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T14:31:39.105 INFO:teuthology.run_tasks:Running task ceph... 2024-04-24T14:31:39.194 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-24T14:31:39.194 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /var/log/ceph 2024-04-24T14:31:39.196 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /var/log/ceph 2024-04-24T14:31:39.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /var/log/ceph 2024-04-24T14:31:39.214 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-24T14:31:39.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T14:31:39.217 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T14:31:39.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T14:31:39.273 INFO:tasks.ceph:Creating extra log directories... 2024-04-24T14:31:39.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T14:31:39.275 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T14:31:39.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T14:31:39.298 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-24T14:31:39.299 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-24T14:31:39.299 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510', 'branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'description': 'rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} thrashers/cache thrashosds-health workloads/rbd_nbd}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7671510', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'], ['client.0']], 'seed': 9954, 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'targets': {'smithi090.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi103.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi173.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create cache 4', 'sudo ceph osd tier add rbd cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay rbd cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 60', 'sudo ceph osd pool set cache target_max_objects 250']}}, {'thrashosds': {'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rbd/rbd-nbd.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_14:09:24', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-24T14:31:39.299 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T14:31:39.301 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T14:31:39.332 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T14:31:39.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T14:31:39.342 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T14:31:39.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T14:31:39.397 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:31:39.397 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T14:31:39.405 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T14:31:39.405 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-04-24T14:31:39.412 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T14:31:39.412 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T14:31:39.412 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 361 Links: 1 2024-04-24T14:31:39.412 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T14:31:39.412 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-24 14:27:49.380737459 +0000 2024-04-24T14:31:39.412 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-24 14:27:49.156742622 +0000 2024-04-24T14:31:39.412 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-24 14:27:49.156742622 +0000 2024-04-24T14:31:39.412 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-04-24T14:31:39.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T14:31:39.427 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-24T14:31:39.428 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-24T14:31:39.428 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000436477 s, 1.2 MB/s 2024-04-24T14:31:39.428 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T14:31:39.437 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-04-24T14:31:39.442 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T14:31:39.442 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T14:31:39.442 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 366 Links: 1 2024-04-24T14:31:39.443 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T14:31:39.443 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-24 14:27:49.568733126 +0000 2024-04-24T14:31:39.443 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-24 14:27:49.564733218 +0000 2024-04-24T14:31:39.443 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-24 14:27:49.564733218 +0000 2024-04-24T14:31:39.443 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-04-24T14:31:39.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T14:31:39.453 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-24T14:31:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-24T14:31:39.454 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000429267 s, 1.2 MB/s 2024-04-24T14:31:39.455 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T14:31:39.507 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-04-24T14:31:39.514 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T14:31:39.514 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T14:31:39.514 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 371 Links: 1 2024-04-24T14:31:39.514 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T14:31:39.514 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-24 14:27:49.984723537 +0000 2024-04-24T14:31:39.514 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-24 14:27:49.980723629 +0000 2024-04-24T14:31:39.514 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-24 14:27:49.980723629 +0000 2024-04-24T14:31:39.514 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-04-24T14:31:39.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T14:31:39.569 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-24T14:31:39.569 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-24T14:31:39.569 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000422588 s, 1.2 MB/s 2024-04-24T14:31:39.571 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T14:31:39.579 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-04-24T14:31:39.585 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T14:31:39.585 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T14:31:39.585 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 379 Links: 1 2024-04-24T14:31:39.585 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T14:31:39.586 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-04-24 14:27:50.620708877 +0000 2024-04-24T14:31:39.586 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-04-24 14:27:50.424713396 +0000 2024-04-24T14:31:39.586 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-04-24 14:27:50.424713396 +0000 2024-04-24T14:31:39.586 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-04-24T14:31:39.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T14:31:39.641 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-04-24T14:31:39.641 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-04-24T14:31:39.641 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000380159 s, 1.3 MB/s 2024-04-24T14:31:39.643 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T14:31:39.650 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T14:31:39.651 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:31:39.651 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T14:31:39.660 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T14:31:39.660 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-04-24T14:31:39.709 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T14:31:39.709 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T14:31:39.709 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 360 Links: 1 2024-04-24T14:31:39.709 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T14:31:39.710 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-24 14:27:49.549078878 +0000 2024-04-24T14:31:39.710 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-24 14:27:49.349083487 +0000 2024-04-24T14:31:39.710 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-24 14:27:49.349083487 +0000 2024-04-24T14:31:39.710 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2024-04-24T14:31:39.710 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T14:31:39.722 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-24T14:31:39.722 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-24T14:31:39.722 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.0004366 s, 1.2 MB/s 2024-04-24T14:31:39.723 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T14:31:39.732 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-04-24T14:31:39.739 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T14:31:39.739 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T14:31:39.739 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 367 Links: 1 2024-04-24T14:31:39.739 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T14:31:39.740 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-24 14:27:50.017068090 +0000 2024-04-24T14:31:39.740 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-24 14:27:49.781073530 +0000 2024-04-24T14:31:39.740 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-24 14:27:49.781073530 +0000 2024-04-24T14:31:39.740 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2024-04-24T14:31:39.740 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T14:31:39.797 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-24T14:31:39.797 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-24T14:31:39.797 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000395971 s, 1.3 MB/s 2024-04-24T14:31:39.798 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T14:31:39.805 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-04-24T14:31:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T14:31:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T14:31:39.810 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 372 Links: 1 2024-04-24T14:31:39.810 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T14:31:39.810 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-24 14:27:50.469057672 +0000 2024-04-24T14:31:39.810 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-24 14:27:50.253062651 +0000 2024-04-24T14:31:39.811 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-24 14:27:50.253062651 +0000 2024-04-24T14:31:39.811 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2024-04-24T14:31:39.811 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T14:31:39.866 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-24T14:31:39.867 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-24T14:31:39.867 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000374712 s, 1.4 MB/s 2024-04-24T14:31:39.868 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T14:31:39.876 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-04-24T14:31:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T14:31:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T14:31:39.882 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 379 Links: 1 2024-04-24T14:31:39.882 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T14:31:39.882 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-04-24 14:27:50.893047899 +0000 2024-04-24T14:31:39.883 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-04-24 14:27:50.677052877 +0000 2024-04-24T14:31:39.883 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-04-24 14:27:50.677052877 +0000 2024-04-24T14:31:39.883 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2024-04-24T14:31:39.883 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T14:31:39.897 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-04-24T14:31:39.897 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-04-24T14:31:39.897 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000336605 s, 1.5 MB/s 2024-04-24T14:31:39.898 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T14:31:39.947 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-24T14:31:39.947 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-24T14:31:39.947 INFO:tasks.ceph:Generating config... 2024-04-24T14:31:39.949 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-24T14:31:39.949 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-24T14:31:39.949 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-24T14:31:39.949 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-24T14:31:39.949 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-24T14:31:39.949 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-24T14:31:39.949 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-24T14:31:39.949 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-24T14:31:39.949 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-04-24T14:31:39.949 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-04-24T14:31:39.949 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-24T14:31:39.950 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-24T14:31:39.950 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-24T14:31:39.950 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-24T14:31:39.950 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-24T14:31:39.950 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-24T14:31:39.950 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-24T14:31:39.950 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-24T14:31:39.950 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-24T14:31:39.950 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-24T14:31:39.950 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-24T14:31:39.950 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-24T14:31:39.950 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-24T14:31:39.951 INFO:tasks.ceph:Setting up mon.a... 2024-04-24T14:31:39.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-24T14:31:39.987 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.keyring 2024-04-24T14:31:39.989 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-24T14:31:40.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T14:31:40.088 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.90'), ('mon.c', '[v2:172.21.15.90:3301,v1:172.21.15.90:6790]'), ('mon.b', '172.21.15.103')] 2024-04-24T14:31:40.088 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.90,[v2:172.21.15.90:3301,v1:172.21.15.90:6790],172.21.15.103', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-24T14:31:40.089 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:31:40.089 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T14:31:40.094 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.90 --addv c '[v2:172.21.15.90:3301,v1:172.21.15.90:6790]' --add b 172.21.15.103 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-24T14:31:40.129 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-24T14:31:40.130 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: generated fsid 6706935e-d09a-472b-b839-321319fa9e54 2024-04-24T14:31:40.130 INFO:teuthology.orchestra.run.smithi090.stdout:setting min_mon_release = octopus 2024-04-24T14:31:40.130 INFO:teuthology.orchestra.run.smithi090.stdout:epoch 0 2024-04-24T14:31:40.130 INFO:teuthology.orchestra.run.smithi090.stdout:fsid 6706935e-d09a-472b-b839-321319fa9e54 2024-04-24T14:31:40.130 INFO:teuthology.orchestra.run.smithi090.stdout:last_changed 2024-04-24T14:31:40.135834+0000 2024-04-24T14:31:40.130 INFO:teuthology.orchestra.run.smithi090.stdout:created 2024-04-24T14:31:40.135834+0000 2024-04-24T14:31:40.130 INFO:teuthology.orchestra.run.smithi090.stdout:min_mon_release 15 (octopus) 2024-04-24T14:31:40.130 INFO:teuthology.orchestra.run.smithi090.stdout:election_strategy: 1 2024-04-24T14:31:40.130 INFO:teuthology.orchestra.run.smithi090.stdout:0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-04-24T14:31:40.130 INFO:teuthology.orchestra.run.smithi090.stdout:1: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-04-24T14:31:40.130 INFO:teuthology.orchestra.run.smithi090.stdout:2: [v2:172.21.15.90:3301/0,v1:172.21.15.90:6790/0] mon.c 2024-04-24T14:31:40.131 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-24T14:31:40.132 DEBUG:teuthology.orchestra.run.smithi090:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T14:31:40.136 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6706935e-d09a-472b-b839-321319fa9e54... 2024-04-24T14:31:40.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T14:31:40.180 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T14:31:40.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T14:31:40.214 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2024-04-24T14:31:40.214 INFO:teuthology.orchestra.run.smithi090.stdout: chdir = "" 2024-04-24T14:31:40.214 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T14:31:40.214 INFO:teuthology.orchestra.run.smithi090.stdout: auth supported = cephx 2024-04-24T14:31:40.214 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.214 INFO:teuthology.orchestra.run.smithi090.stdout: filestore xattr use omap = true 2024-04-24T14:31:40.214 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.214 INFO:teuthology.orchestra.run.smithi090.stdout: mon clock drift allowed = 1.000 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: osd crush chooseleaf type = 0 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: auth debug = true 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on old message = true 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on bug = true 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: mon pg warn max object skew = 0 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default size = 2 2024-04-24T14:31:40.215 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.216 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow primary affinity = true 2024-04-24T14:31:40.216 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow pg remap = true 2024-04-24T14:31:40.216 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on legacy crush tunables = false 2024-04-24T14:31:40.216 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on crush straw calc version zero = false 2024-04-24T14:31:40.216 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on no sortbitwise = false 2024-04-24T14:31:40.216 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on osd down out interval zero = false 2024-04-24T14:31:40.216 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on too few osds = false 2024-04-24T14:31:40.216 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T14:31:40.216 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T14:31:40.216 INFO:teuthology.orchestra.run.smithi090.stdout: mon_allow_pool_size_one = true 2024-04-24T14:31:40.216 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.216 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T14:31:40.216 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.217 INFO:teuthology.orchestra.run.smithi090.stdout: osd default data pool replay window = 5 2024-04-24T14:31:40.217 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.217 INFO:teuthology.orchestra.run.smithi090.stdout: mon allow pool delete = true 2024-04-24T14:31:40.217 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.217 INFO:teuthology.orchestra.run.smithi090.stdout: mon cluster log file level = debug 2024-04-24T14:31:40.217 INFO:teuthology.orchestra.run.smithi090.stdout: debug asserts on shutdown = true 2024-04-24T14:31:40.217 INFO:teuthology.orchestra.run.smithi090.stdout: mon health detail to clog = false 2024-04-24T14:31:40.217 INFO:teuthology.orchestra.run.smithi090.stdout: mon host = "172.21.15.90,[v2:172.21.15.90:3301,v1:172.21.15.90:6790],172.21.15.103" 2024-04-24T14:31:40.217 INFO:teuthology.orchestra.run.smithi090.stdout: mon client directed command retry = 5 2024-04-24T14:31:40.217 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on pool no app = False 2024-04-24T14:31:40.217 INFO:teuthology.orchestra.run.smithi090.stdout: ms inject socket failures = 5000 2024-04-24T14:31:40.218 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = 6706935e-d09a-472b-b839-321319fa9e54 2024-04-24T14:31:40.218 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.218 INFO:teuthology.orchestra.run.smithi090.stdout:[osd] 2024-04-24T14:31:40.218 INFO:teuthology.orchestra.run.smithi090.stdout: osd journal size = 100 2024-04-24T14:31:40.218 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.218 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub load threshold = 5.0 2024-04-24T14:31:40.218 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub max interval = 600 2024-04-24T14:31:40.218 INFO:teuthology.orchestra.run.smithi090.stdout: osd mclock profile = high_recovery_ops 2024-04-24T14:31:40.218 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.218 INFO:teuthology.orchestra.run.smithi090.stdout: osd recover clone overlap = true 2024-04-24T14:31:40.218 INFO:teuthology.orchestra.run.smithi090.stdout: osd recovery max chunk = 1048576 2024-04-24T14:31:40.218 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.218 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug shutdown = true 2024-04-24T14:31:40.219 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug op order = true 2024-04-24T14:31:40.219 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify stray on activate = true 2024-04-24T14:31:40.219 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.219 INFO:teuthology.orchestra.run.smithi090.stdout: osd open classes on start = true 2024-04-24T14:31:40.219 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug pg log writeout = true 2024-04-24T14:31:40.219 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.219 INFO:teuthology.orchestra.run.smithi090.stdout: osd deep scrub update digest min age = 30 2024-04-24T14:31:40.219 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.219 INFO:teuthology.orchestra.run.smithi090.stdout: osd map max advance = 10 2024-04-24T14:31:40.219 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.219 INFO:teuthology.orchestra.run.smithi090.stdout: journal zero on create = true 2024-04-24T14:31:40.219 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.219 INFO:teuthology.orchestra.run.smithi090.stdout: filestore ondisk finisher threads = 3 2024-04-24T14:31:40.219 INFO:teuthology.orchestra.run.smithi090.stdout: filestore apply finisher threads = 3 2024-04-24T14:31:40.220 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.220 INFO:teuthology.orchestra.run.smithi090.stdout: bdev debug aio = true 2024-04-24T14:31:40.220 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug misdirected ops = true 2024-04-24T14:31:40.220 INFO:teuthology.orchestra.run.smithi090.stdout: bluefs allocator = hybrid 2024-04-24T14:31:40.220 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore allocator = hybrid 2024-04-24T14:31:40.220 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore block size = 96636764160 2024-04-24T14:31:40.220 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore fsck on mount = True 2024-04-24T14:31:40.220 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluefs = 1/20 2024-04-24T14:31:40.220 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluestore = 1/20 2024-04-24T14:31:40.220 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-24T14:31:40.220 INFO:teuthology.orchestra.run.smithi090.stdout: debug osd = 20 2024-04-24T14:31:40.220 INFO:teuthology.orchestra.run.smithi090.stdout: debug rocksdb = 4/10 2024-04-24T14:31:40.220 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T14:31:40.220 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd full ratio = 0.9 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd nearfull ratio = 0.8 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout: osd failsafe full ratio = 0.95 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout: osd max markdown count = 1000 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout: osd objectstore = bluestore 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr] 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout: debug mgr = 20 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout: mgr/telemetry/nag = false 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.221 INFO:teuthology.orchestra.run.smithi090.stdout:[mon] 2024-04-24T14:31:40.222 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-04-24T14:31:40.222 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2024-04-24T14:31:40.222 INFO:teuthology.orchestra.run.smithi090.stdout: debug paxos = 20 2024-04-24T14:31:40.222 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2024-04-24T14:31:40.222 INFO:teuthology.orchestra.run.smithi090.stdout: mon data avail warn = 5 2024-04-24T14:31:40.222 INFO:teuthology.orchestra.run.smithi090.stdout: mon mgr mkfs grace = 240 2024-04-24T14:31:40.222 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2024-04-24T14:31:40.222 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd reporter subtree level = osd 2024-04-24T14:31:40.222 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd prime pg temp = true 2024-04-24T14:31:40.222 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2024-04-24T14:31:40.222 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.222 INFO:teuthology.orchestra.run.smithi090.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T14:31:40.222 INFO:teuthology.orchestra.run.smithi090.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T14:31:40.222 INFO:teuthology.orchestra.run.smithi090.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout: # 1m isn't quite enough 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout: mon_down_mkfs_grace = 2m 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_filestore_osds = false 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout:[client] 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout: rgw cache enabled = true 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable ops log = true 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable usage log = true 2024-04-24T14:31:40.223 INFO:teuthology.orchestra.run.smithi090.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T14:31:40.224 INFO:teuthology.orchestra.run.smithi090.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T14:31:40.224 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.a] 2024-04-24T14:31:40.224 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.c] 2024-04-24T14:31:40.224 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.b] 2024-04-24T14:31:40.225 INFO:teuthology.orchestra.run.smithi173.stdout:[global] 2024-04-24T14:31:40.225 INFO:teuthology.orchestra.run.smithi173.stdout: chdir = "" 2024-04-24T14:31:40.225 INFO:teuthology.orchestra.run.smithi173.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T14:31:40.225 INFO:teuthology.orchestra.run.smithi173.stdout: auth supported = cephx 2024-04-24T14:31:40.225 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.225 INFO:teuthology.orchestra.run.smithi173.stdout: filestore xattr use omap = true 2024-04-24T14:31:40.225 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.225 INFO:teuthology.orchestra.run.smithi173.stdout: mon clock drift allowed = 1.000 2024-04-24T14:31:40.225 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.226 INFO:teuthology.orchestra.run.smithi173.stdout: osd crush chooseleaf type = 0 2024-04-24T14:31:40.226 INFO:teuthology.orchestra.run.smithi173.stdout: auth debug = true 2024-04-24T14:31:40.226 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.226 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on old message = true 2024-04-24T14:31:40.226 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on bug = true 2024-04-24T14:31:40.226 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.226 INFO:teuthology.orchestra.run.smithi173.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T14:31:40.226 INFO:teuthology.orchestra.run.smithi173.stdout: mon pg warn max object skew = 0 2024-04-24T14:31:40.226 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.226 INFO:teuthology.orchestra.run.smithi173.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T14:31:40.226 INFO:teuthology.orchestra.run.smithi173.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T14:31:40.226 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.226 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default size = 2 2024-04-24T14:31:40.226 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow primary affinity = true 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow pg remap = true 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on legacy crush tunables = false 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on crush straw calc version zero = false 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on no sortbitwise = false 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on osd down out interval zero = false 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on too few osds = false 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: mon_allow_pool_size_one = true 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: osd default data pool replay window = 5 2024-04-24T14:31:40.227 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.228 INFO:teuthology.orchestra.run.smithi173.stdout: mon allow pool delete = true 2024-04-24T14:31:40.228 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.228 INFO:teuthology.orchestra.run.smithi173.stdout: mon cluster log file level = debug 2024-04-24T14:31:40.228 INFO:teuthology.orchestra.run.smithi173.stdout: debug asserts on shutdown = true 2024-04-24T14:31:40.228 INFO:teuthology.orchestra.run.smithi173.stdout: mon health detail to clog = false 2024-04-24T14:31:40.228 INFO:teuthology.orchestra.run.smithi173.stdout: mon host = "172.21.15.90,[v2:172.21.15.90:3301,v1:172.21.15.90:6790],172.21.15.103" 2024-04-24T14:31:40.228 INFO:teuthology.orchestra.run.smithi173.stdout: mon client directed command retry = 5 2024-04-24T14:31:40.228 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on pool no app = False 2024-04-24T14:31:40.228 INFO:teuthology.orchestra.run.smithi173.stdout: ms inject socket failures = 5000 2024-04-24T14:31:40.228 INFO:teuthology.orchestra.run.smithi173.stdout: fsid = 6706935e-d09a-472b-b839-321319fa9e54 2024-04-24T14:31:40.228 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.228 INFO:teuthology.orchestra.run.smithi173.stdout:[osd] 2024-04-24T14:31:40.228 INFO:teuthology.orchestra.run.smithi173.stdout: osd journal size = 100 2024-04-24T14:31:40.228 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub load threshold = 5.0 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub max interval = 600 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: osd mclock profile = high_recovery_ops 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: osd recover clone overlap = true 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: osd recovery max chunk = 1048576 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug shutdown = true 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug op order = true 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify stray on activate = true 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: osd open classes on start = true 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug pg log writeout = true 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.229 INFO:teuthology.orchestra.run.smithi173.stdout: osd deep scrub update digest min age = 30 2024-04-24T14:31:40.230 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.230 INFO:teuthology.orchestra.run.smithi173.stdout: osd map max advance = 10 2024-04-24T14:31:40.230 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.230 INFO:teuthology.orchestra.run.smithi173.stdout: journal zero on create = true 2024-04-24T14:31:40.230 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.230 INFO:teuthology.orchestra.run.smithi173.stdout: filestore ondisk finisher threads = 3 2024-04-24T14:31:40.230 INFO:teuthology.orchestra.run.smithi173.stdout: filestore apply finisher threads = 3 2024-04-24T14:31:40.230 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.230 INFO:teuthology.orchestra.run.smithi173.stdout: bdev debug aio = true 2024-04-24T14:31:40.230 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug misdirected ops = true 2024-04-24T14:31:40.230 INFO:teuthology.orchestra.run.smithi173.stdout: bluefs allocator = hybrid 2024-04-24T14:31:40.230 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore allocator = hybrid 2024-04-24T14:31:40.230 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore block size = 96636764160 2024-04-24T14:31:40.230 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore fsck on mount = True 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluefs = 1/20 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluestore = 1/20 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout: debug osd = 20 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout: debug rocksdb = 4/10 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd full ratio = 0.9 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd nearfull ratio = 0.8 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout: osd failsafe full ratio = 0.95 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout: osd max markdown count = 1000 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout: osd objectstore = bluestore 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr] 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-04-24T14:31:40.231 INFO:teuthology.orchestra.run.smithi173.stdout: debug mgr = 20 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout: mgr/telemetry/nag = false 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout:[mon] 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout: debug paxos = 20 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout: mon data avail warn = 5 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout: mon mgr mkfs grace = 240 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2024-04-24T14:31:40.232 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd reporter subtree level = osd 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd prime pg temp = true 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: # 1m isn't quite enough 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: mon_down_mkfs_grace = 2m 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.233 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_filestore_osds = false 2024-04-24T14:31:40.234 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-24T14:31:40.234 INFO:teuthology.orchestra.run.smithi173.stdout:[client] 2024-04-24T14:31:40.234 INFO:teuthology.orchestra.run.smithi173.stdout: rgw cache enabled = true 2024-04-24T14:31:40.234 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable ops log = true 2024-04-24T14:31:40.234 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable usage log = true 2024-04-24T14:31:40.234 INFO:teuthology.orchestra.run.smithi173.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T14:31:40.234 INFO:teuthology.orchestra.run.smithi173.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T14:31:40.234 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.a] 2024-04-24T14:31:40.234 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.c] 2024-04-24T14:31:40.234 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.b] 2024-04-24T14:31:40.236 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-04-24T14:31:40.236 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2024-04-24T14:31:40.236 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T14:31:40.236 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2024-04-24T14:31:40.236 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.236 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2024-04-24T14:31:40.236 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.236 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2024-04-24T14:31:40.236 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.236 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2024-04-24T14:31:40.236 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2024-04-24T14:31:40.236 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.236 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2024-04-24T14:31:40.237 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2024-04-24T14:31:40.238 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2024-04-24T14:31:40.238 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2024-04-24T14:31:40.238 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T14:31:40.238 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T14:31:40.238 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2024-04-24T14:31:40.238 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.238 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T14:31:40.238 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.238 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2024-04-24T14:31:40.238 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.238 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2024-04-24T14:31:40.238 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.239 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2024-04-24T14:31:40.239 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2024-04-24T14:31:40.239 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2024-04-24T14:31:40.239 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = "172.21.15.90,[v2:172.21.15.90:3301,v1:172.21.15.90:6790],172.21.15.103" 2024-04-24T14:31:40.239 INFO:teuthology.orchestra.run.smithi103.stdout: mon client directed command retry = 5 2024-04-24T14:31:40.239 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on pool no app = False 2024-04-24T14:31:40.239 INFO:teuthology.orchestra.run.smithi103.stdout: ms inject socket failures = 5000 2024-04-24T14:31:40.239 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = 6706935e-d09a-472b-b839-321319fa9e54 2024-04-24T14:31:40.239 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.239 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2024-04-24T14:31:40.239 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2024-04-24T14:31:40.239 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 600 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: osd mclock profile = high_recovery_ops 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.240 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: bluefs allocator = hybrid 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore allocator = hybrid 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore block size = 96636764160 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: bluestore fsck on mount = True 2024-04-24T14:31:40.241 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluefs = 1/20 2024-04-24T14:31:40.242 INFO:teuthology.orchestra.run.smithi103.stdout: debug bluestore = 1/20 2024-04-24T14:31:40.242 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-24T14:31:40.242 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2024-04-24T14:31:40.242 INFO:teuthology.orchestra.run.smithi103.stdout: debug rocksdb = 4/10 2024-04-24T14:31:40.242 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T14:31:40.242 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd full ratio = 0.9 2024-04-24T14:31:40.242 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd nearfull ratio = 0.8 2024-04-24T14:31:40.242 INFO:teuthology.orchestra.run.smithi103.stdout: osd failsafe full ratio = 0.95 2024-04-24T14:31:40.242 INFO:teuthology.orchestra.run.smithi103.stdout: osd max markdown count = 1000 2024-04-24T14:31:40.242 INFO:teuthology.orchestra.run.smithi103.stdout: osd objectstore = bluestore 2024-04-24T14:31:40.242 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.242 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2024-04-24T14:31:40.242 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2024-04-24T14:31:40.243 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.244 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2024-04-24T14:31:40.245 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-04-24T14:31:40.245 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2024-04-24T14:31:40.245 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2024-04-24T14:31:40.245 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2024-04-24T14:31:40.245 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2024-04-24T14:31:40.245 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T14:31:40.245 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T14:31:40.245 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2024-04-24T14:31:40.245 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.c] 2024-04-24T14:31:40.245 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.b] 2024-04-24T14:31:40.246 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-24T14:31:40.247 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-24T14:31:40.297 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-24T14:31:40.298 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:31:40.298 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-24T14:31:40.304 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:31:40.304 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-24T14:31:40.311 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi090.front.sepia.ceph.com 2024-04-24T14:31:40.311 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:31:40.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T14:31:40.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T14:31:40.334 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:31:40.334 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T14:31:40.385 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T14:31:40.385 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:31:40.385 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T14:31:40.385 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T14:31:40.411 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:31:40.411 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T14:31:40.416 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi173.front.sepia.ceph.com 2024-04-24T14:31:40.416 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-24T14:31:40.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T14:31:40.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T14:31:40.443 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-24T14:31:40.444 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T14:31:40.448 INFO:tasks.ceph:Setting up mon nodes... 2024-04-24T14:31:40.448 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-24T14:31:40.449 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-24T14:31:40.493 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-24T14:31:40.496 INFO:tasks.ceph:Setting up mds nodes... 2024-04-24T14:31:40.496 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-24T14:31:40.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-24T14:31:40.532 INFO:teuthology.orchestra.run.smithi173.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-24T14:31:40.545 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-24T14:31:40.545 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-24T14:31:40.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-24T14:31:40.562 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T14:31:40.562 INFO:tasks.ceph:role: osd.0 2024-04-24T14:31:40.562 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com 2024-04-24T14:31:40.562 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T14:31:41.129 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T14:31:41.129 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T14:31:41.129 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T14:31:41.129 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 2024-04-24T14:31:41.129 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T14:31:41.129 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-04-24T14:31:41.129 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T14:31:41.129 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T14:31:41.129 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T14:31:41.129 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T14:31:41.131 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-04-24T14:31:41.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-24T14:31:41.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-24T14:31:41.247 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T14:31:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T14:31:41.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-24T14:31:41.263 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T14:31:41.264 INFO:tasks.ceph:role: osd.1 2024-04-24T14:31:41.264 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com 2024-04-24T14:31:41.264 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T14:31:41.812 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T14:31:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T14:31:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T14:31:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 2024-04-24T14:31:41.813 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T14:31:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-04-24T14:31:41.813 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T14:31:41.813 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T14:31:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T14:31:41.813 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T14:31:41.815 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-04-24T14:31:41.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-24T14:31:41.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-24T14:31:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T14:31:41.865 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T14:31:41.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-24T14:31:41.875 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T14:31:41.875 INFO:tasks.ceph:role: osd.2 2024-04-24T14:31:41.875 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com 2024-04-24T14:31:41.875 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T14:31:42.360 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T14:31:42.360 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T14:31:42.360 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T14:31:42.360 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 2024-04-24T14:31:42.360 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T14:31:42.360 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-04-24T14:31:42.360 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T14:31:42.360 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T14:31:42.360 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T14:31:42.360 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T14:31:42.362 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-04-24T14:31:42.362 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-24T14:31:42.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-24T14:31:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T14:31:42.433 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T14:31:42.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T14:31:42.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:42.508+0000 7fe5ccbff3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-24T14:31:42.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:42.512+0000 7fe5ccbff3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-24T14:31:42.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:42.512+0000 7fe5ccbff3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T14:31:42.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:42.512+0000 7fe5ccbff3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T14:31:42.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:42.512+0000 7fe5ccbff3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T14:31:42.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:42.512+0000 7fe5ccbff3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-24T14:31:43.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:43.072+0000 7fe5ccbff3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:43.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:43.864+0000 7fe5ccbff3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:44.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:44.644+0000 7fe5ccbff3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:45.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-24T14:31:45.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T14:31:45.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:45.552+0000 7fe490e973c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-24T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:45.552+0000 7fe490e973c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-24T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:45.552+0000 7fe490e973c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:45.552+0000 7fe490e973c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:45.552+0000 7fe490e973c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T14:31:45.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:45.552+0000 7fe490e973c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-24T14:31:46.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:46.116+0000 7fe490e973c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:46.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:46.900+0000 7fe490e973c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:47.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:47.688+0000 7fe490e973c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:48.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-24T14:31:48.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T14:31:48.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:48.604+0000 7f4cdade83c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-24T14:31:48.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:48.604+0000 7f4cdade83c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-24T14:31:48.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:48.604+0000 7f4cdade83c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T14:31:48.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:48.604+0000 7f4cdade83c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T14:31:48.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:48.604+0000 7f4cdade83c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T14:31:48.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:48.604+0000 7f4cdade83c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-24T14:31:49.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:49.160+0000 7f4cdade83c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:49.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:49.952+0000 7f4cdade83c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:50.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:31:50.740+0000 7f4cdade83c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:51.578 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-24T14:31:51.595 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-24T14:31:51.612 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-24T14:31:51.612 INFO:tasks.ceph:role: osd.3 2024-04-24T14:31:51.612 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T14:31:51.612 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T14:31:52.141 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T14:31:52.141 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T14:31:52.141 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T14:31:52.141 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 2024-04-24T14:31:52.141 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T14:31:52.142 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-24T14:31:52.142 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T14:31:52.142 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T14:31:52.142 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T14:31:52.142 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T14:31:52.143 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-24T14:31:52.143 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-04-24T14:31:52.257 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-24T14:31:52.269 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T14:31:52.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T14:31:52.270 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-24T14:31:52.281 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-24T14:31:52.281 INFO:tasks.ceph:role: osd.4 2024-04-24T14:31:52.281 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T14:31:52.281 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T14:31:52.828 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T14:31:52.828 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T14:31:52.829 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T14:31:52.829 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 2024-04-24T14:31:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T14:31:52.829 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-24T14:31:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T14:31:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T14:31:52.829 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T14:31:52.829 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T14:31:52.831 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-24T14:31:52.831 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-04-24T14:31:52.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-24T14:31:52.873 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T14:31:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T14:31:52.874 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-24T14:31:52.883 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-24T14:31:52.883 INFO:tasks.ceph:role: osd.5 2024-04-24T14:31:52.883 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T14:31:52.883 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T14:31:53.417 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T14:31:53.417 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T14:31:53.417 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T14:31:53.418 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 2024-04-24T14:31:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T14:31:53.418 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2024-04-24T14:31:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T14:31:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T14:31:53.418 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T14:31:53.418 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T14:31:53.419 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2024-04-24T14:31:53.419 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-04-24T14:31:53.476 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-24T14:31:53.487 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: /sbin/restorecon: command not found 2024-04-24T14:31:53.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T14:31:53.488 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T14:31:53.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:53.568+0000 7f290751d3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-24T14:31:53.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:53.568+0000 7f290751d3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-24T14:31:53.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:53.568+0000 7f290751d3c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T14:31:53.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:53.568+0000 7f290751d3c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T14:31:53.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:53.568+0000 7f290751d3c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T14:31:53.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:53.572+0000 7f290751d3c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-24T14:31:54.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:54.144+0000 7f290751d3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:54.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:54.916+0000 7f290751d3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:55.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:55.716+0000 7f290751d3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:56.574 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-24T14:31:56.591 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T14:31:56.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:56.680+0000 7f694b9583c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-24T14:31:56.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:56.680+0000 7f694b9583c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-24T14:31:56.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:56.680+0000 7f694b9583c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T14:31:56.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:56.680+0000 7f694b9583c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T14:31:56.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:56.680+0000 7f694b9583c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T14:31:56.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:56.680+0000 7f694b9583c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-24T14:31:57.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:57.232+0000 7f694b9583c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:58.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:58.016+0000 7f694b9583c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:58.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:58.804+0000 7f694b9583c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:31:59.642 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-24T14:31:59.657 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T14:31:59.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:59.696+0000 7f3412bb33c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-24T14:31:59.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:59.696+0000 7f3412bb33c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-24T14:31:59.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:59.696+0000 7f3412bb33c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T14:31:59.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:59.696+0000 7f3412bb33c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T14:31:59.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:59.696+0000 7f3412bb33c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T14:31:59.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:31:59.696+0000 7f3412bb33c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-24T14:32:00.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:32:00.268+0000 7f3412bb33c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:32:01.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:32:01.060+0000 7f3412bb33c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:32:01.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-04-24T14:32:01.840+0000 7f3412bb33c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T14:32:02.693 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-24T14:32:02.708 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-24T14:32:02.708 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:32:02.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-24T14:32:02.726 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:32:02.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-24T14:32:02.741 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:32:02.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-24T14:32:02.802 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:32:02.803 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-24T14:32:02.819 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:32:02.819 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-24T14:32:02.875 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:32:02.875 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-24T14:32:02.885 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:32:02.886 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-24T14:32:02.896 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-24T14:32:02.896 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-24T14:32:02.904 INFO:tasks.ceph:Adding keys to all mons... 2024-04-24T14:32:02.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-24T14:32:02.906 DEBUG:teuthology.orchestra.run.smithi103:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-24T14:32:02.916 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.0] 2024-04-24T14:32:02.917 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDOFylm29yLHhAAGvxCSIBhTuG74AZtVEVs1g== 2024-04-24T14:32:02.917 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.1] 2024-04-24T14:32:02.917 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDRFylm36kGIRAA3amw355qJ3iw/ru6xxHnxQ== 2024-04-24T14:32:02.917 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.2] 2024-04-24T14:32:02.917 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDUFylmUNY7JBAAPdoRq8pFwmcfSpWNzVi1ng== 2024-04-24T14:32:02.917 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.x] 2024-04-24T14:32:02.917 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDMFylmN27eHRAAqdSF2rjQUWV7rjvUW1rRcA== 2024-04-24T14:32:02.917 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.3] 2024-04-24T14:32:02.917 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDZFylmJ5sOIhAAURJyW/R8WYjdp33H9jBLqQ== 2024-04-24T14:32:02.917 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.4] 2024-04-24T14:32:02.917 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDcFylmwGioKBAA5WQ82zYfKc+/8qtBZit5dw== 2024-04-24T14:32:02.917 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.5] 2024-04-24T14:32:02.917 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDfFylm5C61KRAA6HICwDbrguNiht8n113zqw== 2024-04-24T14:32:02.918 INFO:teuthology.orchestra.run.smithi103.stdout:[client.0] 2024-04-24T14:32:02.918 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDMFylmYPkrIBAAs+FZlEepTjtg755nnZty8A== 2024-04-24T14:32:02.919 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.0] 2024-04-24T14:32:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDOFylm29yLHhAAGvxCSIBhTuG74AZtVEVs1g== 2024-04-24T14:32:02.919 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.1] 2024-04-24T14:32:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDRFylm36kGIRAA3amw355qJ3iw/ru6xxHnxQ== 2024-04-24T14:32:02.919 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.2] 2024-04-24T14:32:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDUFylmUNY7JBAAPdoRq8pFwmcfSpWNzVi1ng== 2024-04-24T14:32:02.919 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.x] 2024-04-24T14:32:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDMFylmN27eHRAAqdSF2rjQUWV7rjvUW1rRcA== 2024-04-24T14:32:02.919 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.3] 2024-04-24T14:32:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDZFylmJ5sOIhAAURJyW/R8WYjdp33H9jBLqQ== 2024-04-24T14:32:02.919 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.4] 2024-04-24T14:32:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDcFylmwGioKBAA5WQ82zYfKc+/8qtBZit5dw== 2024-04-24T14:32:02.920 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.5] 2024-04-24T14:32:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDfFylm5C61KRAA6HICwDbrguNiht8n113zqw== 2024-04-24T14:32:02.920 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-04-24T14:32:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDMFylmYPkrIBAAs+FZlEepTjtg755nnZty8A== 2024-04-24T14:32:02.921 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T14:32:02.964 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T14:32:03.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T14:32:03.012 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T14:32:03.056 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T14:32:03.059 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T14:32:03.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T14:32:03.146 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T14:32:03.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T14:32:03.192 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T14:32:03.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T14:32:03.236 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T14:32:03.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T14:32:03.321 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T14:32:03.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T14:32:03.367 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T14:32:03.414 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-24T14:32:03.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-24T14:32:03.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T14:32:03.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-24T14:32:03.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-24T14:32:03.558 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T14:32:03.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-24T14:32:03.618 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-24T14:32:03.636 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T14:32:03.690 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-24T14:32:03.702 DEBUG:teuthology.orchestra.run.smithi090:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-24T14:32:03.704 DEBUG:teuthology.orchestra.run.smithi103:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-24T14:32:03.709 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-24T14:32:03.709 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-24T14:32:03.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-24T14:32:03.711 INFO:tasks.ceph.mon.a:Started 2024-04-24T14:32:03.712 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-24T14:32:03.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-24T14:32:03.714 INFO:tasks.ceph.mon.c:Started 2024-04-24T14:32:03.714 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-24T14:32:03.715 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-24T14:32:03.756 INFO:tasks.ceph.mon.b:Started 2024-04-24T14:32:03.756 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-24T14:32:03.756 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-24T14:32:03.756 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-24T14:32:03.758 INFO:tasks.ceph.mgr.x:Started 2024-04-24T14:32:03.759 INFO:tasks.ceph:Setting crush tunables to default 2024-04-24T14:32:03.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd crush tunables default 2024-04-24T14:32:09.268 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:09.272+0000 7f9197598000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T14:32:09.895 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:09.900+0000 7f9197598000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T14:32:09.951 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:09.956+0000 7f9197598000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T14:32:09.960 INFO:teuthology.orchestra.run.smithi090.stderr:adjusted tunables profile to default 2024-04-24T14:32:09.971 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-24T14:32:09.971 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:32:09.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-24T14:32:09.981 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:32:09.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-24T14:32:09.991 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-04-24T14:32:09.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-24T14:32:10.004 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:32:10.004 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-24T14:32:10.018 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:32:10.018 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-24T14:32:10.065 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:10.068+0000 7f9197598000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T14:32:10.077 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-04-24T14:32:10.077 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-24T14:32:10.087 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new a2506b4f-3486-4f51-972f-1d9e4b015346 0 2024-04-24T14:32:10.283 INFO:teuthology.orchestra.run.smithi103.stdout:0 2024-04-24T14:32:10.289 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:10.292+0000 7f9197598000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T14:32:10.294 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 9e17324f-a615-494f-ba0d-7c8aa8dac95c 1 2024-04-24T14:32:10.338 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:10.340+0000 7f9197598000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T14:32:10.488 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-04-24T14:32:10.500 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7 2 2024-04-24T14:32:10.518 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:10.520+0000 7f9197598000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T14:32:10.580 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:10.584+0000 7f9197598000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T14:32:10.641 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:10.644+0000 7f9197598000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T14:32:10.716 INFO:teuthology.orchestra.run.smithi103.stdout:2 2024-04-24T14:32:10.729 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 509c16da-22b2-48e8-aa51-c320c8c4fd8b 3 2024-04-24T14:32:10.942 INFO:teuthology.orchestra.run.smithi103.stdout:3 2024-04-24T14:32:10.955 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 1462066b-4e59-46d4-9b85-1f61ee1d56a6 4 2024-04-24T14:32:11.064 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:11.068+0000 7f9197598000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T14:32:11.144 INFO:teuthology.orchestra.run.smithi103.stdout:4 2024-04-24T14:32:11.156 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 4dc954a8-5f3c-4826-b9bc-98cf30519382 5 2024-04-24T14:32:11.175 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:11.180+0000 7f9197598000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T14:32:11.350 INFO:teuthology.orchestra.run.smithi103.stdout:5 2024-04-24T14:32:11.362 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-24T14:32:11.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-24T14:32:11.365 INFO:tasks.ceph.osd.0:Started 2024-04-24T14:32:11.365 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-24T14:32:11.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-24T14:32:11.367 INFO:tasks.ceph.osd.1:Started 2024-04-24T14:32:11.367 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-24T14:32:11.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-24T14:32:11.370 INFO:tasks.ceph.osd.2:Started 2024-04-24T14:32:11.370 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-24T14:32:11.370 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-24T14:32:11.372 INFO:tasks.ceph.osd.3:Started 2024-04-24T14:32:11.372 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-24T14:32:11.372 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-24T14:32:11.374 INFO:tasks.ceph.osd.4:Started 2024-04-24T14:32:11.374 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-24T14:32:11.374 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-24T14:32:11.376 INFO:tasks.ceph.osd.5:Started 2024-04-24T14:32:11.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T14:32:11.483 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:11.488+0000 7f9197598000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T14:32:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:11.546 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":8,"fsid":"6706935e-d09a-472b-b839-321319fa9e54","created":"2024-04-24T14:32:09.059925+0000","modified":"2024-04-24T14:32:11.347028+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T14:32:11.347028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"a2506b4f-3486-4f51-972f-1d9e4b015346","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9e17324f-a615-494f-ba0d-7c8aa8dac95c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"509c16da-22b2-48e8-aa51-c320c8c4fd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1462066b-4e59-46d4-9b85-1f61ee1d56a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4dc954a8-5f3c-4826-b9bc-98cf30519382","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T14:32:11.557 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-24T14:32:11.558 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-24T14:32:11.645 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:11.648+0000 7f9197598000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T14:32:11.710 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:11.712+0000 7f9197598000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T14:32:11.716 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:11.720+0000 7f713d7923c0 -1 Falling back to public interface 2024-04-24T14:32:11.731 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:11.736+0000 7fe866bf23c0 -1 Falling back to public interface 2024-04-24T14:32:11.733 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:11.735+0000 7f2f47b323c0 -1 Falling back to public interface 2024-04-24T14:32:11.761 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:11.764+0000 7f9197598000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T14:32:11.816 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:11.820+0000 7f9197598000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T14:32:11.972 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:11.976+0000 7f5487a133c0 -1 Falling back to public interface 2024-04-24T14:32:11.974 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:11.976+0000 7f9197598000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T14:32:11.985 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:11.987+0000 7ff18ca123c0 -1 Falling back to public interface 2024-04-24T14:32:12.005 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:12.007+0000 7f6704b923c0 -1 Falling back to public interface 2024-04-24T14:32:12.140 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:12.144+0000 7f9197598000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T14:32:12.162 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T14:32:12.206 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:12.208+0000 7f9197598000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T14:32:12.343 INFO:teuthology.misc.health.smithi090.stdout: 2024-04-24T14:32:12.343 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":8,"fsid":"6706935e-d09a-472b-b839-321319fa9e54","created":"2024-04-24T14:32:09.059925+0000","modified":"2024-04-24T14:32:11.347028+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T14:32:11.347028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"a2506b4f-3486-4f51-972f-1d9e4b015346","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9e17324f-a615-494f-ba0d-7c8aa8dac95c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"509c16da-22b2-48e8-aa51-c320c8c4fd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1462066b-4e59-46d4-9b85-1f61ee1d56a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4dc954a8-5f3c-4826-b9bc-98cf30519382","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T14:32:12.353 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-04-24T14:32:12.367 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:12.372+0000 7f9197598000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T14:32:12.434 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:12.436+0000 7f9197598000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T14:32:13.128 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:13.132+0000 7f9197598000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T14:32:14.171 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:14.176+0000 7f9180650700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-24T14:32:14.171 INFO:tasks.ceph.mgr.x.smithi103.stderr:2024-04-24T14:32:14.176+0000 7f9180650700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-24T14:32:15.536 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:15.540+0000 7fe866bf23c0 -1 osd.4 0 log_to_monitors true 2024-04-24T14:32:15.539 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:15.543+0000 7f2f47b323c0 -1 osd.1 0 log_to_monitors true 2024-04-24T14:32:15.609 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:15.612+0000 7f713d7923c0 -1 osd.3 0 log_to_monitors true 2024-04-24T14:32:15.704 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:15.707+0000 7ff18ca123c0 -1 osd.2 0 log_to_monitors true 2024-04-24T14:32:15.721 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:15.724+0000 7f5487a133c0 -1 osd.5 0 log_to_monitors true 2024-04-24T14:32:15.817 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:15.819+0000 7f6704b923c0 -1 osd.0 0 log_to_monitors true 2024-04-24T14:32:18.959 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T14:32:19.205 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:19.207+0000 7ff1881b2700 -1 osd.2 0 waiting for initial osdmap 2024-04-24T14:32:19.214 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:19.215+0000 7ff180f55700 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T14:32:19.221 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:19.224+0000 7fe863bb7700 -1 osd.4 0 waiting for initial osdmap 2024-04-24T14:32:19.226 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:19.232+0000 7fe85b135700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T14:32:19.238 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:19.244+0000 7f54849d8700 -1 osd.5 0 waiting for initial osdmap 2024-04-24T14:32:19.245 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:19.248+0000 7f547bf56700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T14:32:19.266 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:19.267+0000 7f2f432d2700 -1 osd.1 0 waiting for initial osdmap 2024-04-24T14:32:19.273 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:19.275+0000 7f2f3c075700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T14:32:19.275 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:19.280+0000 7f713a757700 -1 osd.3 0 waiting for initial osdmap 2024-04-24T14:32:19.287 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:19.292+0000 7f7131cd5700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T14:32:19.325 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:19.327+0000 7f6700332700 -1 osd.0 0 waiting for initial osdmap 2024-04-24T14:32:19.338 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:19.339+0000 7f66f90d5700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T14:32:19.364 INFO:teuthology.misc.health.smithi090.stdout: 2024-04-24T14:32:19.364 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":10,"fsid":"6706935e-d09a-472b-b839-321319fa9e54","created":"2024-04-24T14:32:09.059925+0000","modified":"2024-04-24T14:32:17.176979+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T14:32:11.347028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"a2506b4f-3486-4f51-972f-1d9e4b015346","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9e17324f-a615-494f-ba0d-7c8aa8dac95c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"509c16da-22b2-48e8-aa51-c320c8c4fd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1462066b-4e59-46d4-9b85-1f61ee1d56a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4dc954a8-5f3c-4826-b9bc-98cf30519382","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T14:32:19.379 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-04-24T14:32:25.986 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T14:32:26.266 INFO:teuthology.misc.health.smithi090.stdout: 2024-04-24T14:32:26.267 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":14,"fsid":"6706935e-d09a-472b-b839-321319fa9e54","created":"2024-04-24T14:32:09.059925+0000","modified":"2024-04-24T14:32:23.269544+0000","last_up_change":"2024-04-24T14:32:20.236258+0000","last_in_change":"2024-04-24T14:32:11.347028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T14:32:21.233022+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a2506b4f-3486-4f51-972f-1d9e4b015346","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":20309},{"type":"v1","addr":"172.21.15.90:6817","nonce":20309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":20309},{"type":"v1","addr":"172.21.15.90:6819","nonce":20309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":20309},{"type":"v1","addr":"172.21.15.90:6823","nonce":20309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":20309},{"type":"v1","addr":"172.21.15.90:6821","nonce":20309}]},"public_addr":"172.21.15.90:6817/20309","cluster_addr":"172.21.15.90:6819/20309","heartbeat_back_addr":"172.21.15.90:6823/20309","heartbeat_front_addr":"172.21.15.90:6821/20309","state":["exists","up"]},{"osd":1,"uuid":"9e17324f-a615-494f-ba0d-7c8aa8dac95c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":20308},{"type":"v1","addr":"172.21.15.90:6801","nonce":20308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":20308},{"type":"v1","addr":"172.21.15.90:6803","nonce":20308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":20308},{"type":"v1","addr":"172.21.15.90:6807","nonce":20308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":20308},{"type":"v1","addr":"172.21.15.90:6805","nonce":20308}]},"public_addr":"172.21.15.90:6801/20308","cluster_addr":"172.21.15.90:6803/20308","heartbeat_back_addr":"172.21.15.90:6807/20308","heartbeat_front_addr":"172.21.15.90:6805/20308","state":["exists","up"]},{"osd":2,"uuid":"f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":20310},{"type":"v1","addr":"172.21.15.90:6809","nonce":20310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":20310},{"type":"v1","addr":"172.21.15.90:6811","nonce":20310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":20310},{"type":"v1","addr":"172.21.15.90:6815","nonce":20310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":20310},{"type":"v1","addr":"172.21.15.90:6813","nonce":20310}]},"public_addr":"172.21.15.90:6809/20310","cluster_addr":"172.21.15.90:6811/20310","heartbeat_back_addr":"172.21.15.90:6815/20310","heartbeat_front_addr":"172.21.15.90:6813/20310","state":["exists","up"]},{"osd":3,"uuid":"509c16da-22b2-48e8-aa51-c320c8c4fd8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":20316},{"type":"v1","addr":"172.21.15.103:6801","nonce":20316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":20316},{"type":"v1","addr":"172.21.15.103:6803","nonce":20316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":20316},{"type":"v1","addr":"172.21.15.103:6807","nonce":20316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":20316},{"type":"v1","addr":"172.21.15.103:6805","nonce":20316}]},"public_addr":"172.21.15.103:6801/20316","cluster_addr":"172.21.15.103:6803/20316","heartbeat_back_addr":"172.21.15.103:6807/20316","heartbeat_front_addr":"172.21.15.103:6805/20316","state":["exists","up"]},{"osd":4,"uuid":"1462066b-4e59-46d4-9b85-1f61ee1d56a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":20318},{"type":"v1","addr":"172.21.15.103:6809","nonce":20318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":20318},{"type":"v1","addr":"172.21.15.103:6811","nonce":20318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":20318},{"type":"v1","addr":"172.21.15.103:6815","nonce":20318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":20318},{"type":"v1","addr":"172.21.15.103:6813","nonce":20318}]},"public_addr":"172.21.15.103:6809/20318","cluster_addr":"172.21.15.103:6811/20318","heartbeat_back_addr":"172.21.15.103:6815/20318","heartbeat_front_addr":"172.21.15.103:6813/20318","state":["exists","up"]},{"osd":5,"uuid":"4dc954a8-5f3c-4826-b9bc-98cf30519382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":20317},{"type":"v1","addr":"172.21.15.103:6817","nonce":20317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":20317},{"type":"v1","addr":"172.21.15.103:6819","nonce":20317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":20317},{"type":"v1","addr":"172.21.15.103:6823","nonce":20317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":20317},{"type":"v1","addr":"172.21.15.103:6821","nonce":20317}]},"public_addr":"172.21.15.103:6817/20317","cluster_addr":"172.21.15.103:6819/20317","heartbeat_back_addr":"172.21.15.103:6823/20317","heartbeat_front_addr":"172.21.15.103:6821/20317","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.792899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.580414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.713336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.579638+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.570510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.736558+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T14:32:26.279 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-04-24T14:32:26.279 INFO:tasks.ceph:Creating RBD pool 2024-04-24T14:32:26.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-24T14:32:27.297 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'rbd' created 2024-04-24T14:32:27.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-24T14:32:28.313 INFO:teuthology.orchestra.run.smithi090.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-24T14:32:28.325 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-24T14:32:28.326 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-24T14:32:28.326 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-24T14:32:28.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-24T14:32:28.326 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-24T14:32:28.652 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:28.664 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":20070},{"type":"v1","addr":"172.21.15.103:6825","nonce":20070}]},"active_addr":"172.21.15.103:6825/20070","active_change":"2024-04-24T14:32:13.155404+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":4205697526}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":764388718}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":1367638240}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":335335895}]}]} 2024-04-24T14:32:28.669 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-24T14:32:28.669 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-24T14:32:28.669 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T14:32:28.953 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:28.953 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":16,"fsid":"6706935e-d09a-472b-b839-321319fa9e54","created":"2024-04-24T14:32:09.059925+0000","modified":"2024-04-24T14:32:28.291252+0000","last_up_change":"2024-04-24T14:32:20.236258+0000","last_in_change":"2024-04-24T14:32:11.347028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T14:32:21.233022+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T14:32:26.609964+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a2506b4f-3486-4f51-972f-1d9e4b015346","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":20309},{"type":"v1","addr":"172.21.15.90:6817","nonce":20309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":20309},{"type":"v1","addr":"172.21.15.90:6819","nonce":20309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":20309},{"type":"v1","addr":"172.21.15.90:6823","nonce":20309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":20309},{"type":"v1","addr":"172.21.15.90:6821","nonce":20309}]},"public_addr":"172.21.15.90:6817/20309","cluster_addr":"172.21.15.90:6819/20309","heartbeat_back_addr":"172.21.15.90:6823/20309","heartbeat_front_addr":"172.21.15.90:6821/20309","state":["exists","up"]},{"osd":1,"uuid":"9e17324f-a615-494f-ba0d-7c8aa8dac95c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":20308},{"type":"v1","addr":"172.21.15.90:6801","nonce":20308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":20308},{"type":"v1","addr":"172.21.15.90:6803","nonce":20308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":20308},{"type":"v1","addr":"172.21.15.90:6807","nonce":20308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":20308},{"type":"v1","addr":"172.21.15.90:6805","nonce":20308}]},"public_addr":"172.21.15.90:6801/20308","cluster_addr":"172.21.15.90:6803/20308","heartbeat_back_addr":"172.21.15.90:6807/20308","heartbeat_front_addr":"172.21.15.90:6805/20308","state":["exists","up"]},{"osd":2,"uuid":"f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":20310},{"type":"v1","addr":"172.21.15.90:6809","nonce":20310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":20310},{"type":"v1","addr":"172.21.15.90:6811","nonce":20310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":20310},{"type":"v1","addr":"172.21.15.90:6815","nonce":20310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":20310},{"type":"v1","addr":"172.21.15.90:6813","nonce":20310}]},"public_addr":"172.21.15.90:6809/20310","cluster_addr":"172.21.15.90:6811/20310","heartbeat_back_addr":"172.21.15.90:6815/20310","heartbeat_front_addr":"172.21.15.90:6813/20310","state":["exists","up"]},{"osd":3,"uuid":"509c16da-22b2-48e8-aa51-c320c8c4fd8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":20316},{"type":"v1","addr":"172.21.15.103:6801","nonce":20316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":20316},{"type":"v1","addr":"172.21.15.103:6803","nonce":20316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":20316},{"type":"v1","addr":"172.21.15.103:6807","nonce":20316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":20316},{"type":"v1","addr":"172.21.15.103:6805","nonce":20316}]},"public_addr":"172.21.15.103:6801/20316","cluster_addr":"172.21.15.103:6803/20316","heartbeat_back_addr":"172.21.15.103:6807/20316","heartbeat_front_addr":"172.21.15.103:6805/20316","state":["exists","up"]},{"osd":4,"uuid":"1462066b-4e59-46d4-9b85-1f61ee1d56a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":20318},{"type":"v1","addr":"172.21.15.103:6809","nonce":20318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":20318},{"type":"v1","addr":"172.21.15.103:6811","nonce":20318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":20318},{"type":"v1","addr":"172.21.15.103:6815","nonce":20318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":20318},{"type":"v1","addr":"172.21.15.103:6813","nonce":20318}]},"public_addr":"172.21.15.103:6809/20318","cluster_addr":"172.21.15.103:6811/20318","heartbeat_back_addr":"172.21.15.103:6815/20318","heartbeat_front_addr":"172.21.15.103:6813/20318","state":["exists","up"]},{"osd":5,"uuid":"4dc954a8-5f3c-4826-b9bc-98cf30519382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":20317},{"type":"v1","addr":"172.21.15.103:6817","nonce":20317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":20317},{"type":"v1","addr":"172.21.15.103:6819","nonce":20317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":20317},{"type":"v1","addr":"172.21.15.103:6823","nonce":20317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":20317},{"type":"v1","addr":"172.21.15.103:6821","nonce":20317}]},"public_addr":"172.21.15.103:6817/20317","cluster_addr":"172.21.15.103:6819/20317","heartbeat_back_addr":"172.21.15.103:6823/20317","heartbeat_front_addr":"172.21.15.103:6821/20317","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.792899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.580414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.713336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.579638+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.570510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.736558+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T14:32:28.966 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-24T14:32:28.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T14:32:29.254 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:29.255 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":16,"fsid":"6706935e-d09a-472b-b839-321319fa9e54","created":"2024-04-24T14:32:09.059925+0000","modified":"2024-04-24T14:32:28.291252+0000","last_up_change":"2024-04-24T14:32:20.236258+0000","last_in_change":"2024-04-24T14:32:11.347028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T14:32:21.233022+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T14:32:26.609964+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a2506b4f-3486-4f51-972f-1d9e4b015346","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":20309},{"type":"v1","addr":"172.21.15.90:6817","nonce":20309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":20309},{"type":"v1","addr":"172.21.15.90:6819","nonce":20309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":20309},{"type":"v1","addr":"172.21.15.90:6823","nonce":20309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":20309},{"type":"v1","addr":"172.21.15.90:6821","nonce":20309}]},"public_addr":"172.21.15.90:6817/20309","cluster_addr":"172.21.15.90:6819/20309","heartbeat_back_addr":"172.21.15.90:6823/20309","heartbeat_front_addr":"172.21.15.90:6821/20309","state":["exists","up"]},{"osd":1,"uuid":"9e17324f-a615-494f-ba0d-7c8aa8dac95c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":20308},{"type":"v1","addr":"172.21.15.90:6801","nonce":20308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":20308},{"type":"v1","addr":"172.21.15.90:6803","nonce":20308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":20308},{"type":"v1","addr":"172.21.15.90:6807","nonce":20308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":20308},{"type":"v1","addr":"172.21.15.90:6805","nonce":20308}]},"public_addr":"172.21.15.90:6801/20308","cluster_addr":"172.21.15.90:6803/20308","heartbeat_back_addr":"172.21.15.90:6807/20308","heartbeat_front_addr":"172.21.15.90:6805/20308","state":["exists","up"]},{"osd":2,"uuid":"f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":20310},{"type":"v1","addr":"172.21.15.90:6809","nonce":20310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":20310},{"type":"v1","addr":"172.21.15.90:6811","nonce":20310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":20310},{"type":"v1","addr":"172.21.15.90:6815","nonce":20310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":20310},{"type":"v1","addr":"172.21.15.90:6813","nonce":20310}]},"public_addr":"172.21.15.90:6809/20310","cluster_addr":"172.21.15.90:6811/20310","heartbeat_back_addr":"172.21.15.90:6815/20310","heartbeat_front_addr":"172.21.15.90:6813/20310","state":["exists","up"]},{"osd":3,"uuid":"509c16da-22b2-48e8-aa51-c320c8c4fd8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":20316},{"type":"v1","addr":"172.21.15.103:6801","nonce":20316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":20316},{"type":"v1","addr":"172.21.15.103:6803","nonce":20316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":20316},{"type":"v1","addr":"172.21.15.103:6807","nonce":20316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":20316},{"type":"v1","addr":"172.21.15.103:6805","nonce":20316}]},"public_addr":"172.21.15.103:6801/20316","cluster_addr":"172.21.15.103:6803/20316","heartbeat_back_addr":"172.21.15.103:6807/20316","heartbeat_front_addr":"172.21.15.103:6805/20316","state":["exists","up"]},{"osd":4,"uuid":"1462066b-4e59-46d4-9b85-1f61ee1d56a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":20318},{"type":"v1","addr":"172.21.15.103:6809","nonce":20318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":20318},{"type":"v1","addr":"172.21.15.103:6811","nonce":20318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":20318},{"type":"v1","addr":"172.21.15.103:6815","nonce":20318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":20318},{"type":"v1","addr":"172.21.15.103:6813","nonce":20318}]},"public_addr":"172.21.15.103:6809/20318","cluster_addr":"172.21.15.103:6811/20318","heartbeat_back_addr":"172.21.15.103:6815/20318","heartbeat_front_addr":"172.21.15.103:6813/20318","state":["exists","up"]},{"osd":5,"uuid":"4dc954a8-5f3c-4826-b9bc-98cf30519382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":20317},{"type":"v1","addr":"172.21.15.103:6817","nonce":20317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":20317},{"type":"v1","addr":"172.21.15.103:6819","nonce":20317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":20317},{"type":"v1","addr":"172.21.15.103:6823","nonce":20317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":20317},{"type":"v1","addr":"172.21.15.103:6821","nonce":20317}]},"public_addr":"172.21.15.103:6817/20317","cluster_addr":"172.21.15.103:6819/20317","heartbeat_back_addr":"172.21.15.103:6823/20317","heartbeat_front_addr":"172.21.15.103:6821/20317","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.792899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.580414+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.713336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.579638+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.570510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.736558+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T14:32:29.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-24T14:32:29.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-24T14:32:29.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-24T14:32:29.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-24T14:32:29.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-24T14:32:29.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-24T14:32:29.424 INFO:teuthology.orchestra.run.smithi090.stdout:47244640262 2024-04-24T14:32:29.425 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T14:32:29.432 INFO:teuthology.orchestra.run.smithi090.stdout:47244640262 2024-04-24T14:32:29.432 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T14:32:29.448 INFO:teuthology.orchestra.run.smithi090.stdout:47244640261 2024-04-24T14:32:29.449 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T14:32:29.450 INFO:teuthology.orchestra.run.smithi090.stdout:47244640262 2024-04-24T14:32:29.450 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T14:32:29.458 INFO:teuthology.orchestra.run.smithi090.stdout:47244640260 2024-04-24T14:32:29.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T14:32:29.463 INFO:teuthology.orchestra.run.smithi090.stdout:47244640262 2024-04-24T14:32:29.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T14:32:29.775 INFO:teuthology.orchestra.run.smithi090.stdout:47244640261 2024-04-24T14:32:29.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.1 2024-04-24T14:32:29.809 INFO:teuthology.orchestra.run.smithi090.stdout:47244640261 2024-04-24T14:32:29.821 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.2 2024-04-24T14:32:29.841 INFO:teuthology.orchestra.run.smithi090.stdout:47244640261 2024-04-24T14:32:29.860 INFO:teuthology.orchestra.run.smithi090.stdout:47244640261 2024-04-24T14:32:29.861 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.5 2024-04-24T14:32:29.872 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.4 2024-04-24T14:32:29.879 INFO:teuthology.orchestra.run.smithi090.stdout:47244640260 2024-04-24T14:32:29.894 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.0 2024-04-24T14:32:29.921 INFO:teuthology.orchestra.run.smithi090.stdout:47244640259 2024-04-24T14:32:29.934 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.3 2024-04-24T14:32:30.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T14:32:30.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T14:32:30.862 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T14:32:30.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T14:32:30.895 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T14:32:30.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T14:32:31.155 INFO:teuthology.orchestra.run.smithi090.stdout:47244640261 2024-04-24T14:32:31.160 INFO:teuthology.orchestra.run.smithi090.stdout:47244640261 2024-04-24T14:32:31.168 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.1 2024-04-24T14:32:31.173 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.2 2024-04-24T14:32:31.284 INFO:teuthology.orchestra.run.smithi090.stdout:47244640261 2024-04-24T14:32:31.295 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.4 2024-04-24T14:32:31.296 INFO:teuthology.orchestra.run.smithi090.stdout:47244640260 2024-04-24T14:32:31.296 INFO:teuthology.orchestra.run.smithi090.stdout:47244640261 2024-04-24T14:32:31.309 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.0 2024-04-24T14:32:31.313 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.5 2024-04-24T14:32:31.338 INFO:teuthology.orchestra.run.smithi090.stdout:47244640260 2024-04-24T14:32:31.350 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.3 2024-04-24T14:32:31.350 DEBUG:teuthology.parallel:result is None 2024-04-24T14:32:32.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T14:32:32.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T14:32:32.296 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T14:32:32.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T14:32:32.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T14:32:32.534 INFO:teuthology.orchestra.run.smithi090.stdout:47244640262 2024-04-24T14:32:32.550 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.1 2024-04-24T14:32:32.550 DEBUG:teuthology.parallel:result is None 2024-04-24T14:32:32.563 INFO:teuthology.orchestra.run.smithi090.stdout:47244640262 2024-04-24T14:32:32.575 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.2 2024-04-24T14:32:32.576 DEBUG:teuthology.parallel:result is None 2024-04-24T14:32:32.650 INFO:teuthology.orchestra.run.smithi090.stdout:47244640262 2024-04-24T14:32:32.663 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.4 2024-04-24T14:32:32.664 DEBUG:teuthology.parallel:result is None 2024-04-24T14:32:32.671 INFO:teuthology.orchestra.run.smithi090.stdout:47244640261 2024-04-24T14:32:32.677 INFO:teuthology.orchestra.run.smithi090.stdout:47244640262 2024-04-24T14:32:32.684 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.0 2024-04-24T14:32:32.685 DEBUG:teuthology.parallel:result is None 2024-04-24T14:32:32.690 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.5 2024-04-24T14:32:32.690 DEBUG:teuthology.parallel:result is None 2024-04-24T14:32:32.690 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T14:32:32.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:32:32.979 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:32.981 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:32:32.992 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-24T14:32:31.172810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1790272,"kb_used_data":2024,"kb_used_omap":9,"kb_used_meta":1788150,"kb_avail":564440768,"statfs":{"total":579820584960,"available":577987346432,"internally_reserved":0,"allocated":2072576,"data_stored":1460966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1831065616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001076"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.325541+0000","last_change":"2024-04-24T14:32:28.321025+0000","last_active":"2024-04-24T14:32:28.325541+0000","last_peered":"2024-04-24T14:32:28.325541+0000","last_clean":"2024-04-24T14:32:28.325541+0000","last_became_active":"2024-04-24T14:32:28.320369+0000","last_became_peered":"2024-04-24T14:32:28.320369+0000","last_unstale":"2024-04-24T14:32:28.325541+0000","last_undegraded":"2024-04-24T14:32:28.325541+0000","last_fullsized":"2024-04-24T14:32:28.325541+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:49:07.246787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.317301+0000","last_change":"2024-04-24T14:32:28.317301+0000","last_active":"2024-04-24T14:32:28.317301+0000","last_peered":"2024-04-24T14:32:28.317301+0000","last_clean":"2024-04-24T14:32:28.317301+0000","last_became_active":"2024-04-24T14:32:28.316717+0000","last_became_peered":"2024-04-24T14:32:28.316717+0000","last_unstale":"2024-04-24T14:32:28.317301+0000","last_undegraded":"2024-04-24T14:32:28.317301+0000","last_fullsized":"2024-04-24T14:32:28.317301+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:58:17.830985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.325588+0000","last_change":"2024-04-24T14:32:28.321917+0000","last_active":"2024-04-24T14:32:28.325588+0000","last_peered":"2024-04-24T14:32:28.325588+0000","last_clean":"2024-04-24T14:32:28.325588+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:32:28.325588+0000","last_undegraded":"2024-04-24T14:32:28.325588+0000","last_fullsized":"2024-04-24T14:32:28.325588+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.319745+0000","last_change":"2024-04-24T14:32:28.319745+0000","last_active":"2024-04-24T14:32:28.319745+0000","last_peered":"2024-04-24T14:32:28.319745+0000","last_clean":"2024-04-24T14:32:28.319745+0000","last_became_active":"2024-04-24T14:32:28.319444+0000","last_became_peered":"2024-04-24T14:32:28.319444+0000","last_unstale":"2024-04-24T14:32:28.319745+0000","last_undegraded":"2024-04-24T14:32:28.319745+0000","last_fullsized":"2024-04-24T14:32:28.319745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:21:01.144715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.321633+0000","last_change":"2024-04-24T14:32:28.321633+0000","last_active":"2024-04-24T14:32:28.321633+0000","last_peered":"2024-04-24T14:32:28.321633+0000","last_clean":"2024-04-24T14:32:28.321633+0000","last_became_active":"2024-04-24T14:32:28.321107+0000","last_became_peered":"2024-04-24T14:32:28.321107+0000","last_unstale":"2024-04-24T14:32:28.321633+0000","last_undegraded":"2024-04-24T14:32:28.321633+0000","last_fullsized":"2024-04-24T14:32:28.321633+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:57.639201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.321535+0000","last_change":"2024-04-24T14:32:28.316625+0000","last_active":"2024-04-24T14:32:28.321535+0000","last_peered":"2024-04-24T14:32:28.321535+0000","last_clean":"2024-04-24T14:32:28.321535+0000","last_became_active":"2024-04-24T14:32:28.315716+0000","last_became_peered":"2024-04-24T14:32:28.315716+0000","last_unstale":"2024-04-24T14:32:28.321535+0000","last_undegraded":"2024-04-24T14:32:28.321535+0000","last_fullsized":"2024-04-24T14:32:28.321535+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:48:30.673644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.325683+0000","last_change":"2024-04-24T14:32:28.322591+0000","last_active":"2024-04-24T14:32:28.325683+0000","last_peered":"2024-04-24T14:32:28.325683+0000","last_clean":"2024-04-24T14:32:28.325683+0000","last_became_active":"2024-04-24T14:32:28.321701+0000","last_became_peered":"2024-04-24T14:32:28.321701+0000","last_unstale":"2024-04-24T14:32:28.325683+0000","last_undegraded":"2024-04-24T14:32:28.325683+0000","last_fullsized":"2024-04-24T14:32:28.325683+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:56:22.511442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.322438+0000","last_change":"2024-04-24T14:32:28.322438+0000","last_active":"2024-04-24T14:32:28.322438+0000","last_peered":"2024-04-24T14:32:28.322438+0000","last_clean":"2024-04-24T14:32:28.322438+0000","last_became_active":"2024-04-24T14:32:28.321824+0000","last_became_peered":"2024-04-24T14:32:28.321824+0000","last_unstale":"2024-04-24T14:32:28.322438+0000","last_undegraded":"2024-04-24T14:32:28.322438+0000","last_fullsized":"2024-04-24T14:32:28.322438+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:26:37.494259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.314885+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:32:28.314885+0000","last_peered":"2024-04-24T14:32:28.314885+0000","last_clean":"2024-04-24T14:32:28.314885+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:32:28.314885+0000","last_undegraded":"2024-04-24T14:32:28.314885+0000","last_fullsized":"2024-04-24T14:32:28.314885+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":46705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]}]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":46705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302176,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94069664,"statfs":{"total":96636764160,"available":96327335936,"internally_reserved":0,"allocated":741376,"data_stored":637073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]}]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":46705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":147456,"data_stored":46705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298084,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073756,"statfs":{"total":96636764160,"available":96331526144,"internally_reserved":0,"allocated":741376,"data_stored":637073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:32:32.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:32:33.271 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:33.274 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:32:33.284 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-04-24T14:32:33.173469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1790352,"kb_used_data":2096,"kb_used_omap":9,"kb_used_meta":1788150,"kb_avail":564440688,"statfs":{"total":579820584960,"available":577987264512,"internally_reserved":0,"allocated":2146304,"data_stored":1490198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1831065616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001735"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.325541+0000","last_change":"2024-04-24T14:32:28.321025+0000","last_active":"2024-04-24T14:32:28.325541+0000","last_peered":"2024-04-24T14:32:28.325541+0000","last_clean":"2024-04-24T14:32:28.325541+0000","last_became_active":"2024-04-24T14:32:28.320369+0000","last_became_peered":"2024-04-24T14:32:28.320369+0000","last_unstale":"2024-04-24T14:32:28.325541+0000","last_undegraded":"2024-04-24T14:32:28.325541+0000","last_fullsized":"2024-04-24T14:32:28.325541+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:49:07.246787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.321816+0000","last_change":"2024-04-24T14:32:28.317301+0000","last_active":"2024-04-24T14:32:28.321816+0000","last_peered":"2024-04-24T14:32:28.321816+0000","last_clean":"2024-04-24T14:32:28.321816+0000","last_became_active":"2024-04-24T14:32:28.316717+0000","last_became_peered":"2024-04-24T14:32:28.316717+0000","last_unstale":"2024-04-24T14:32:28.321816+0000","last_undegraded":"2024-04-24T14:32:28.321816+0000","last_fullsized":"2024-04-24T14:32:28.321816+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:58:17.830985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.325588+0000","last_change":"2024-04-24T14:32:28.321917+0000","last_active":"2024-04-24T14:32:28.325588+0000","last_peered":"2024-04-24T14:32:28.325588+0000","last_clean":"2024-04-24T14:32:28.325588+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:32:28.325588+0000","last_undegraded":"2024-04-24T14:32:28.325588+0000","last_fullsized":"2024-04-24T14:32:28.325588+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.325340+0000","last_change":"2024-04-24T14:32:28.319745+0000","last_active":"2024-04-24T14:32:28.325340+0000","last_peered":"2024-04-24T14:32:28.325340+0000","last_clean":"2024-04-24T14:32:28.325340+0000","last_became_active":"2024-04-24T14:32:28.319444+0000","last_became_peered":"2024-04-24T14:32:28.319444+0000","last_unstale":"2024-04-24T14:32:28.325340+0000","last_undegraded":"2024-04-24T14:32:28.325340+0000","last_fullsized":"2024-04-24T14:32:28.325340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:21:01.144715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.325368+0000","last_change":"2024-04-24T14:32:28.321633+0000","last_active":"2024-04-24T14:32:28.325368+0000","last_peered":"2024-04-24T14:32:28.325368+0000","last_clean":"2024-04-24T14:32:28.325368+0000","last_became_active":"2024-04-24T14:32:28.321107+0000","last_became_peered":"2024-04-24T14:32:28.321107+0000","last_unstale":"2024-04-24T14:32:28.325368+0000","last_undegraded":"2024-04-24T14:32:28.325368+0000","last_fullsized":"2024-04-24T14:32:28.325368+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:57.639201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.321535+0000","last_change":"2024-04-24T14:32:28.316625+0000","last_active":"2024-04-24T14:32:28.321535+0000","last_peered":"2024-04-24T14:32:28.321535+0000","last_clean":"2024-04-24T14:32:28.321535+0000","last_became_active":"2024-04-24T14:32:28.315716+0000","last_became_peered":"2024-04-24T14:32:28.315716+0000","last_unstale":"2024-04-24T14:32:28.321535+0000","last_undegraded":"2024-04-24T14:32:28.321535+0000","last_fullsized":"2024-04-24T14:32:28.321535+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:48:30.673644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.325683+0000","last_change":"2024-04-24T14:32:28.322591+0000","last_active":"2024-04-24T14:32:28.325683+0000","last_peered":"2024-04-24T14:32:28.325683+0000","last_clean":"2024-04-24T14:32:28.325683+0000","last_became_active":"2024-04-24T14:32:28.321701+0000","last_became_peered":"2024-04-24T14:32:28.321701+0000","last_unstale":"2024-04-24T14:32:28.325683+0000","last_undegraded":"2024-04-24T14:32:28.325683+0000","last_fullsized":"2024-04-24T14:32:28.325683+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:56:22.511442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.325392+0000","last_change":"2024-04-24T14:32:28.322438+0000","last_active":"2024-04-24T14:32:28.325392+0000","last_peered":"2024-04-24T14:32:28.325392+0000","last_clean":"2024-04-24T14:32:28.325392+0000","last_became_active":"2024-04-24T14:32:28.321824+0000","last_became_peered":"2024-04-24T14:32:28.321824+0000","last_unstale":"2024-04-24T14:32:28.325392+0000","last_undegraded":"2024-04-24T14:32:28.325392+0000","last_fullsized":"2024-04-24T14:32:28.325392+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:26:37.494259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-24T14:32:28.314885+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:32:28.314885+0000","last_peered":"2024-04-24T14:32:28.314885+0000","last_clean":"2024-04-24T14:32:28.314885+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:32:28.314885+0000","last_undegraded":"2024-04-24T14:32:28.314885+0000","last_fullsized":"2024-04-24T14:32:28.314885+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640263,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":159744,"data_stored":51577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]}]},{"osd":4,"up_from":11,"seq":47244640263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":159744,"data_stored":51577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302192,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94069648,"statfs":{"total":96636764160,"available":96327319552,"internally_reserved":0,"allocated":753664,"data_stored":641945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]}]},{"osd":2,"up_from":11,"seq":47244640263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297520,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074320,"statfs":{"total":96636764160,"available":96332103680,"internally_reserved":0,"allocated":159744,"data_stored":51577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":51577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298100,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073740,"statfs":{"total":96636764160,"available":96331509760,"internally_reserved":0,"allocated":753664,"data_stored":641945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:32:33.285 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T14:32:33.285 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-24T14:32:33.285 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-24T14:32:33.286 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T14:32:33.602 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:33.602 INFO:teuthology.orchestra.run.smithi090.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T14:32:33.620 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-24T14:32:33.620 INFO:teuthology.run_tasks:Running task exec... 2024-04-24T14:32:33.630 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T14:32:33.631 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi173.front.sepia.ceph.com 2024-04-24T14:32:33.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create cache 4' 2024-04-24T14:32:34.344 INFO:teuthology.orchestra.run.smithi173.stderr:pool 'cache' created 2024-04-24T14:32:34.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier add rbd cache' 2024-04-24T14:32:35.352 INFO:teuthology.orchestra.run.smithi173.stderr:pool 'cache' is now (or already was) a tier of 'rbd' 2024-04-24T14:32:35.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier cache-mode cache writeback' 2024-04-24T14:32:36.363 INFO:teuthology.orchestra.run.smithi173.stderr:set cache-mode for pool 'cache' to writeback 2024-04-24T14:32:36.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier set-overlay rbd cache' 2024-04-24T14:32:37.366 INFO:teuthology.orchestra.run.smithi173.stderr:overlay for 'rbd' is now (or already was) 'cache' 2024-04-24T14:32:37.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_type bloom' 2024-04-24T14:32:38.375 INFO:teuthology.orchestra.run.smithi173.stderr:set pool 3 hit_set_type to bloom 2024-04-24T14:32:38.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_count 8' 2024-04-24T14:32:39.386 INFO:teuthology.orchestra.run.smithi173.stderr:set pool 3 hit_set_count to 8 2024-04-24T14:32:39.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_period 60' 2024-04-24T14:32:40.388 INFO:teuthology.orchestra.run.smithi173.stderr:set pool 3 hit_set_period to 60 2024-04-24T14:32:40.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache target_max_objects 250' 2024-04-24T14:32:41.407 INFO:teuthology.orchestra.run.smithi173.stderr:set pool 3 target_max_objects to 250 2024-04-24T14:32:41.424 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-24T14:32:41.439 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-24T14:32:41.440 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}} 2024-04-24T14:32:41.440 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-04-24T14:32:41.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-24T14:32:41.767 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-24T14:32:41.767 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T14:32:41.767 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:32:42.048 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:42.051 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:32:42.062 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-04-24T14:32:41.400733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1794904,"kb_used_data":2564,"kb_used_omap":9,"kb_used_meta":1792246,"kb_avail":564436136,"statfs":{"total":579820584960,"available":577982603264,"internally_reserved":0,"allocated":2625536,"data_stored":1699481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1835259920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.035831"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:39.386721+0000","last_change":"2024-04-24T14:32:28.321025+0000","last_active":"2024-04-24T14:32:39.386721+0000","last_peered":"2024-04-24T14:32:39.386721+0000","last_clean":"2024-04-24T14:32:39.386721+0000","last_became_active":"2024-04-24T14:32:28.320369+0000","last_became_peered":"2024-04-24T14:32:28.320369+0000","last_unstale":"2024-04-24T14:32:39.386721+0000","last_undegraded":"2024-04-24T14:32:39.386721+0000","last_fullsized":"2024-04-24T14:32:39.386721+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:49:07.246787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:40.040018+0000","last_change":"2024-04-24T14:32:28.317301+0000","last_active":"2024-04-24T14:32:40.040018+0000","last_peered":"2024-04-24T14:32:40.040018+0000","last_clean":"2024-04-24T14:32:40.040018+0000","last_became_active":"2024-04-24T14:32:28.316717+0000","last_became_peered":"2024-04-24T14:32:28.316717+0000","last_unstale":"2024-04-24T14:32:40.040018+0000","last_undegraded":"2024-04-24T14:32:40.040018+0000","last_fullsized":"2024-04-24T14:32:40.040018+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:58:17.830985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:39.386699+0000","last_change":"2024-04-24T14:32:28.321917+0000","last_active":"2024-04-24T14:32:39.386699+0000","last_peered":"2024-04-24T14:32:39.386699+0000","last_clean":"2024-04-24T14:32:39.386699+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:32:39.386699+0000","last_undegraded":"2024-04-24T14:32:39.386699+0000","last_fullsized":"2024-04-24T14:32:39.386699+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:40.040111+0000","last_change":"2024-04-24T14:32:28.319745+0000","last_active":"2024-04-24T14:32:40.040111+0000","last_peered":"2024-04-24T14:32:40.040111+0000","last_clean":"2024-04-24T14:32:40.040111+0000","last_became_active":"2024-04-24T14:32:28.319444+0000","last_became_peered":"2024-04-24T14:32:28.319444+0000","last_unstale":"2024-04-24T14:32:40.040111+0000","last_undegraded":"2024-04-24T14:32:40.040111+0000","last_fullsized":"2024-04-24T14:32:40.040111+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:21:01.144715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-24T14:32:39.539944+0000","last_change":"2024-04-24T14:32:35.360857+0000","last_active":"2024-04-24T14:32:39.539944+0000","last_peered":"2024-04-24T14:32:39.539944+0000","last_clean":"2024-04-24T14:32:39.539944+0000","last_became_active":"2024-04-24T14:32:35.360300+0000","last_became_peered":"2024-04-24T14:32:35.360300+0000","last_unstale":"2024-04-24T14:32:39.539944+0000","last_undegraded":"2024-04-24T14:32:39.539944+0000","last_fullsized":"2024-04-24T14:32:39.539944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:16:49.472230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-24T14:32:37.367625+0000","last_change":"2024-04-24T14:32:28.321633+0000","last_active":"2024-04-24T14:32:37.367625+0000","last_peered":"2024-04-24T14:32:37.367625+0000","last_clean":"2024-04-24T14:32:37.367625+0000","last_became_active":"2024-04-24T14:32:28.321107+0000","last_became_peered":"2024-04-24T14:32:28.321107+0000","last_unstale":"2024-04-24T14:32:37.367625+0000","last_undegraded":"2024-04-24T14:32:37.367625+0000","last_fullsized":"2024-04-24T14:32:37.367625+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:57.639201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:40.039718+0000","last_change":"2024-04-24T14:32:35.358803+0000","last_active":"2024-04-24T14:32:40.039718+0000","last_peered":"2024-04-24T14:32:40.039718+0000","last_clean":"2024-04-24T14:32:40.039718+0000","last_became_active":"2024-04-24T14:32:35.358235+0000","last_became_peered":"2024-04-24T14:32:35.358235+0000","last_unstale":"2024-04-24T14:32:40.039718+0000","last_undegraded":"2024-04-24T14:32:40.039718+0000","last_fullsized":"2024-04-24T14:32:40.039718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:38:39.388528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-24T14:32:38.373949+0000","last_change":"2024-04-24T14:32:28.316625+0000","last_active":"2024-04-24T14:32:38.373949+0000","last_peered":"2024-04-24T14:32:38.373949+0000","last_clean":"2024-04-24T14:32:38.373949+0000","last_became_active":"2024-04-24T14:32:28.315716+0000","last_became_peered":"2024-04-24T14:32:28.315716+0000","last_unstale":"2024-04-24T14:32:38.373949+0000","last_undegraded":"2024-04-24T14:32:38.373949+0000","last_fullsized":"2024-04-24T14:32:38.373949+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:48:30.673644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:39.383046+0000","last_change":"2024-04-24T14:32:35.360501+0000","last_active":"2024-04-24T14:32:39.383046+0000","last_peered":"2024-04-24T14:32:39.383046+0000","last_clean":"2024-04-24T14:32:39.383046+0000","last_became_active":"2024-04-24T14:32:35.359880+0000","last_became_peered":"2024-04-24T14:32:35.359880+0000","last_unstale":"2024-04-24T14:32:39.383046+0000","last_undegraded":"2024-04-24T14:32:39.383046+0000","last_fullsized":"2024-04-24T14:32:39.383046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:09:50.332041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:39.386700+0000","last_change":"2024-04-24T14:32:28.322591+0000","last_active":"2024-04-24T14:32:39.386700+0000","last_peered":"2024-04-24T14:32:39.386700+0000","last_clean":"2024-04-24T14:32:39.386700+0000","last_became_active":"2024-04-24T14:32:28.321701+0000","last_became_peered":"2024-04-24T14:32:28.321701+0000","last_unstale":"2024-04-24T14:32:39.386700+0000","last_undegraded":"2024-04-24T14:32:39.386700+0000","last_fullsized":"2024-04-24T14:32:39.386700+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:56:22.511442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:40.039883+0000","last_change":"2024-04-24T14:32:35.821258+0000","last_active":"2024-04-24T14:32:40.039883+0000","last_peered":"2024-04-24T14:32:40.039883+0000","last_clean":"2024-04-24T14:32:40.039883+0000","last_became_active":"2024-04-24T14:32:35.820625+0000","last_became_peered":"2024-04-24T14:32:35.820625+0000","last_unstale":"2024-04-24T14:32:40.039883+0000","last_undegraded":"2024-04-24T14:32:40.039883+0000","last_fullsized":"2024-04-24T14:32:40.039883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:42:30.996596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-24T14:32:37.367637+0000","last_change":"2024-04-24T14:32:28.322438+0000","last_active":"2024-04-24T14:32:37.367637+0000","last_peered":"2024-04-24T14:32:37.367637+0000","last_clean":"2024-04-24T14:32:37.367637+0000","last_became_active":"2024-04-24T14:32:28.321824+0000","last_became_peered":"2024-04-24T14:32:28.321824+0000","last_unstale":"2024-04-24T14:32:37.367637+0000","last_undegraded":"2024-04-24T14:32:37.367637+0000","last_fullsized":"2024-04-24T14:32:37.367637+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:26:37.494259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":168,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:39.386706+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:32:39.386706+0000","last_peered":"2024-04-24T14:32:39.386706+0000","last_clean":"2024-04-24T14:32:39.386706+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:32:39.386706+0000","last_undegraded":"2024-04-24T14:32:39.386706+0000","last_fullsized":"2024-04-24T14:32:39.386706+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297584,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074256,"statfs":{"total":96636764160,"available":96332038144,"internally_reserved":0,"allocated":233472,"data_stored":83814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297588,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074252,"statfs":{"total":96636764160,"available":96332034048,"internally_reserved":0,"allocated":233472,"data_stored":83814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]}]},{"osd":3,"up_from":11,"seq":47244640263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302272,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94069568,"statfs":{"total":96636764160,"available":96327237632,"internally_reserved":0,"allocated":839680,"data_stored":679469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]}]},{"osd":2,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297584,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074256,"statfs":{"total":96636764160,"available":96332038144,"internally_reserved":0,"allocated":233472,"data_stored":83814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]}]},{"osd":1,"up_from":11,"seq":47244640265,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297600,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074240,"statfs":{"total":96636764160,"available":96332021760,"internally_reserved":0,"allocated":245760,"data_stored":89101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640264,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302276,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94069564,"statfs":{"total":96636764160,"available":96327233536,"internally_reserved":0,"allocated":839680,"data_stored":679469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:32:42.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:32:42.363 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:42.366 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:32:42.377 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-04-24T14:32:41.400733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1794904,"kb_used_data":2564,"kb_used_omap":9,"kb_used_meta":1792246,"kb_avail":564436136,"statfs":{"total":579820584960,"available":577982603264,"internally_reserved":0,"allocated":2625536,"data_stored":1699481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1835259920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.035831"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:39.386721+0000","last_change":"2024-04-24T14:32:28.321025+0000","last_active":"2024-04-24T14:32:39.386721+0000","last_peered":"2024-04-24T14:32:39.386721+0000","last_clean":"2024-04-24T14:32:39.386721+0000","last_became_active":"2024-04-24T14:32:28.320369+0000","last_became_peered":"2024-04-24T14:32:28.320369+0000","last_unstale":"2024-04-24T14:32:39.386721+0000","last_undegraded":"2024-04-24T14:32:39.386721+0000","last_fullsized":"2024-04-24T14:32:39.386721+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:49:07.246787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:40.040018+0000","last_change":"2024-04-24T14:32:28.317301+0000","last_active":"2024-04-24T14:32:40.040018+0000","last_peered":"2024-04-24T14:32:40.040018+0000","last_clean":"2024-04-24T14:32:40.040018+0000","last_became_active":"2024-04-24T14:32:28.316717+0000","last_became_peered":"2024-04-24T14:32:28.316717+0000","last_unstale":"2024-04-24T14:32:40.040018+0000","last_undegraded":"2024-04-24T14:32:40.040018+0000","last_fullsized":"2024-04-24T14:32:40.040018+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:58:17.830985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:39.386699+0000","last_change":"2024-04-24T14:32:28.321917+0000","last_active":"2024-04-24T14:32:39.386699+0000","last_peered":"2024-04-24T14:32:39.386699+0000","last_clean":"2024-04-24T14:32:39.386699+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:32:39.386699+0000","last_undegraded":"2024-04-24T14:32:39.386699+0000","last_fullsized":"2024-04-24T14:32:39.386699+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:40.040111+0000","last_change":"2024-04-24T14:32:28.319745+0000","last_active":"2024-04-24T14:32:40.040111+0000","last_peered":"2024-04-24T14:32:40.040111+0000","last_clean":"2024-04-24T14:32:40.040111+0000","last_became_active":"2024-04-24T14:32:28.319444+0000","last_became_peered":"2024-04-24T14:32:28.319444+0000","last_unstale":"2024-04-24T14:32:40.040111+0000","last_undegraded":"2024-04-24T14:32:40.040111+0000","last_fullsized":"2024-04-24T14:32:40.040111+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:21:01.144715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-24T14:32:39.539944+0000","last_change":"2024-04-24T14:32:35.360857+0000","last_active":"2024-04-24T14:32:39.539944+0000","last_peered":"2024-04-24T14:32:39.539944+0000","last_clean":"2024-04-24T14:32:39.539944+0000","last_became_active":"2024-04-24T14:32:35.360300+0000","last_became_peered":"2024-04-24T14:32:35.360300+0000","last_unstale":"2024-04-24T14:32:39.539944+0000","last_undegraded":"2024-04-24T14:32:39.539944+0000","last_fullsized":"2024-04-24T14:32:39.539944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:16:49.472230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-24T14:32:37.367625+0000","last_change":"2024-04-24T14:32:28.321633+0000","last_active":"2024-04-24T14:32:37.367625+0000","last_peered":"2024-04-24T14:32:37.367625+0000","last_clean":"2024-04-24T14:32:37.367625+0000","last_became_active":"2024-04-24T14:32:28.321107+0000","last_became_peered":"2024-04-24T14:32:28.321107+0000","last_unstale":"2024-04-24T14:32:37.367625+0000","last_undegraded":"2024-04-24T14:32:37.367625+0000","last_fullsized":"2024-04-24T14:32:37.367625+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:57.639201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:40.039718+0000","last_change":"2024-04-24T14:32:35.358803+0000","last_active":"2024-04-24T14:32:40.039718+0000","last_peered":"2024-04-24T14:32:40.039718+0000","last_clean":"2024-04-24T14:32:40.039718+0000","last_became_active":"2024-04-24T14:32:35.358235+0000","last_became_peered":"2024-04-24T14:32:35.358235+0000","last_unstale":"2024-04-24T14:32:40.039718+0000","last_undegraded":"2024-04-24T14:32:40.039718+0000","last_fullsized":"2024-04-24T14:32:40.039718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:38:39.388528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-24T14:32:38.373949+0000","last_change":"2024-04-24T14:32:28.316625+0000","last_active":"2024-04-24T14:32:38.373949+0000","last_peered":"2024-04-24T14:32:38.373949+0000","last_clean":"2024-04-24T14:32:38.373949+0000","last_became_active":"2024-04-24T14:32:28.315716+0000","last_became_peered":"2024-04-24T14:32:28.315716+0000","last_unstale":"2024-04-24T14:32:38.373949+0000","last_undegraded":"2024-04-24T14:32:38.373949+0000","last_fullsized":"2024-04-24T14:32:38.373949+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:48:30.673644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:39.383046+0000","last_change":"2024-04-24T14:32:35.360501+0000","last_active":"2024-04-24T14:32:39.383046+0000","last_peered":"2024-04-24T14:32:39.383046+0000","last_clean":"2024-04-24T14:32:39.383046+0000","last_became_active":"2024-04-24T14:32:35.359880+0000","last_became_peered":"2024-04-24T14:32:35.359880+0000","last_unstale":"2024-04-24T14:32:39.383046+0000","last_undegraded":"2024-04-24T14:32:39.383046+0000","last_fullsized":"2024-04-24T14:32:39.383046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:09:50.332041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:39.386700+0000","last_change":"2024-04-24T14:32:28.322591+0000","last_active":"2024-04-24T14:32:39.386700+0000","last_peered":"2024-04-24T14:32:39.386700+0000","last_clean":"2024-04-24T14:32:39.386700+0000","last_became_active":"2024-04-24T14:32:28.321701+0000","last_became_peered":"2024-04-24T14:32:28.321701+0000","last_unstale":"2024-04-24T14:32:39.386700+0000","last_undegraded":"2024-04-24T14:32:39.386700+0000","last_fullsized":"2024-04-24T14:32:39.386700+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:56:22.511442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:40.039883+0000","last_change":"2024-04-24T14:32:35.821258+0000","last_active":"2024-04-24T14:32:40.039883+0000","last_peered":"2024-04-24T14:32:40.039883+0000","last_clean":"2024-04-24T14:32:40.039883+0000","last_became_active":"2024-04-24T14:32:35.820625+0000","last_became_peered":"2024-04-24T14:32:35.820625+0000","last_unstale":"2024-04-24T14:32:40.039883+0000","last_undegraded":"2024-04-24T14:32:40.039883+0000","last_fullsized":"2024-04-24T14:32:40.039883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:42:30.996596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-24T14:32:37.367637+0000","last_change":"2024-04-24T14:32:28.322438+0000","last_active":"2024-04-24T14:32:37.367637+0000","last_peered":"2024-04-24T14:32:37.367637+0000","last_clean":"2024-04-24T14:32:37.367637+0000","last_became_active":"2024-04-24T14:32:28.321824+0000","last_became_peered":"2024-04-24T14:32:28.321824+0000","last_unstale":"2024-04-24T14:32:37.367637+0000","last_undegraded":"2024-04-24T14:32:37.367637+0000","last_fullsized":"2024-04-24T14:32:37.367637+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:26:37.494259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":168,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-24T14:32:39.386706+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:32:39.386706+0000","last_peered":"2024-04-24T14:32:39.386706+0000","last_clean":"2024-04-24T14:32:39.386706+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:32:39.386706+0000","last_undegraded":"2024-04-24T14:32:39.386706+0000","last_fullsized":"2024-04-24T14:32:39.386706+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297584,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074256,"statfs":{"total":96636764160,"available":96332038144,"internally_reserved":0,"allocated":233472,"data_stored":83814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297588,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074252,"statfs":{"total":96636764160,"available":96332034048,"internally_reserved":0,"allocated":233472,"data_stored":83814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]}]},{"osd":3,"up_from":11,"seq":47244640263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302272,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94069568,"statfs":{"total":96636764160,"available":96327237632,"internally_reserved":0,"allocated":839680,"data_stored":679469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]}]},{"osd":2,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297584,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074256,"statfs":{"total":96636764160,"available":96332038144,"internally_reserved":0,"allocated":233472,"data_stored":83814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]}]},{"osd":1,"up_from":11,"seq":47244640265,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297600,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074240,"statfs":{"total":96636764160,"available":96332021760,"internally_reserved":0,"allocated":245760,"data_stored":89101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640264,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302276,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94069564,"statfs":{"total":96636764160,"available":96327233536,"internally_reserved":0,"allocated":839680,"data_stored":679469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:32:42.378 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T14:32:42.378 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-24T14:32:42.667 INFO:teuthology.orchestra.run.smithi090.stdout:epoch 25 2024-04-24T14:32:42.667 INFO:teuthology.orchestra.run.smithi090.stdout:fsid 6706935e-d09a-472b-b839-321319fa9e54 2024-04-24T14:32:42.667 INFO:teuthology.orchestra.run.smithi090.stdout:created 2024-04-24T14:32:09.059925+0000 2024-04-24T14:32:42.667 INFO:teuthology.orchestra.run.smithi090.stdout:modified 2024-04-24T14:32:41.389351+0000 2024-04-24T14:32:42.667 INFO:teuthology.orchestra.run.smithi090.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-24T14:32:42.667 INFO:teuthology.orchestra.run.smithi090.stdout:crush_version 4 2024-04-24T14:32:42.668 INFO:teuthology.orchestra.run.smithi090.stdout:full_ratio 0.95 2024-04-24T14:32:42.668 INFO:teuthology.orchestra.run.smithi090.stdout:backfillfull_ratio 0.9 2024-04-24T14:32:42.668 INFO:teuthology.orchestra.run.smithi090.stdout:nearfull_ratio 0.85 2024-04-24T14:32:42.668 INFO:teuthology.orchestra.run.smithi090.stdout:require_min_compat_client luminous 2024-04-24T14:32:42.668 INFO:teuthology.orchestra.run.smithi090.stdout:min_compat_client jewel 2024-04-24T14:32:42.668 INFO:teuthology.orchestra.run.smithi090.stdout:require_osd_release quincy 2024-04-24T14:32:42.668 INFO:teuthology.orchestra.run.smithi090.stdout:stretch_mode_enabled false 2024-04-24T14:32:42.668 INFO:teuthology.orchestra.run.smithi090.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 14 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2024-04-24T14:32:42.668 INFO:teuthology.orchestra.run.smithi090.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 21 lfor 21/21/21 flags hashpspool tiers 3 read_tier 3 write_tier 3 stripe_width 0 application rbd 2024-04-24T14:32:42.668 INFO:teuthology.orchestra.run.smithi090.stdout:pool 3 'cache' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 25 lfor 21/21/21 flags hashpspool,incomplete_clones tier_of 2 cache_mode writeback target_objects 250 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 60s x8 decay_rate 0 search_last_n 0 stripe_width 0 application rbd 2024-04-24T14:32:42.668 INFO:teuthology.orchestra.run.smithi090.stdout:max_osd 6 2024-04-24T14:32:42.668 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0 up in weight 1 up_from 11 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6816/20309,v1:172.21.15.90:6817/20309] [v2:172.21.15.90:6818/20309,v1:172.21.15.90:6819/20309] exists,up a2506b4f-3486-4f51-972f-1d9e4b015346 2024-04-24T14:32:42.668 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1 up in weight 1 up_from 11 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6800/20308,v1:172.21.15.90:6801/20308] [v2:172.21.15.90:6802/20308,v1:172.21.15.90:6803/20308] exists,up 9e17324f-a615-494f-ba0d-7c8aa8dac95c 2024-04-24T14:32:42.668 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6808/20310,v1:172.21.15.90:6809/20310] [v2:172.21.15.90:6810/20310,v1:172.21.15.90:6811/20310] exists,up f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7 2024-04-24T14:32:42.669 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6800/20316,v1:172.21.15.103:6801/20316] [v2:172.21.15.103:6802/20316,v1:172.21.15.103:6803/20316] exists,up 509c16da-22b2-48e8-aa51-c320c8c4fd8b 2024-04-24T14:32:42.669 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4 up in weight 1 up_from 11 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6808/20318,v1:172.21.15.103:6809/20318] [v2:172.21.15.103:6810/20318,v1:172.21.15.103:6811/20318] exists,up 1462066b-4e59-46d4-9b85-1f61ee1d56a6 2024-04-24T14:32:42.669 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6816/20317,v1:172.21.15.103:6817/20317] [v2:172.21.15.103:6818/20317,v1:172.21.15.103:6819/20317] exists,up 4dc954a8-5f3c-4826-b9bc-98cf30519382 2024-04-24T14:32:42.681 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 11 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6816/20309,v1:172.21.15.90:6817/20309] [v2:172.21.15.90:6818/20309,v1:172.21.15.90:6819/20309] exists,up a2506b4f-3486-4f51-972f-1d9e4b015346', 'osd.1 up in weight 1 up_from 11 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6800/20308,v1:172.21.15.90:6801/20308] [v2:172.21.15.90:6802/20308,v1:172.21.15.90:6803/20308] exists,up 9e17324f-a615-494f-ba0d-7c8aa8dac95c', 'osd.2 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6808/20310,v1:172.21.15.90:6809/20310] [v2:172.21.15.90:6810/20310,v1:172.21.15.90:6811/20310] exists,up f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7', 'osd.3 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6800/20316,v1:172.21.15.103:6801/20316] [v2:172.21.15.103:6802/20316,v1:172.21.15.103:6803/20316] exists,up 509c16da-22b2-48e8-aa51-c320c8c4fd8b', 'osd.4 up in weight 1 up_from 11 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6808/20318,v1:172.21.15.103:6809/20318] [v2:172.21.15.103:6810/20318,v1:172.21.15.103:6811/20318] exists,up 1462066b-4e59-46d4-9b85-1f61ee1d56a6', 'osd.5 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6816/20317,v1:172.21.15.103:6817/20317] [v2:172.21.15.103:6818/20317,v1:172.21.15.103:6819/20317] exists,up 4dc954a8-5f3c-4826-b9bc-98cf30519382'] 2024-04-24T14:32:42.681 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-24T14:32:42.776 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:42.777 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mon.a", 2024-04-24T14:32:42.777 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster": "ceph", 2024-04-24T14:32:42.777 INFO:teuthology.orchestra.run.smithi090.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-24T14:32:42.777 INFO:teuthology.orchestra.run.smithi090.stdout: "admin_socket_mode": "", 2024-04-24T14:32:42.777 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-24T14:32:42.777 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_client_required": "cephx, none", 2024-04-24T14:32:42.777 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_cluster_required": "cephx", 2024-04-24T14:32:42.777 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_debug": "true", 2024-04-24T14:32:42.777 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-24T14:32:42.777 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-24T14:32:42.777 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_service_required": "cephx", 2024-04-24T14:32:42.777 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-24T14:32:42.778 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_supported": "cephx", 2024-04-24T14:32:42.778 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_aio": "true", 2024-04-24T14:32:42.778 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-24T14:32:42.778 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_aio_poll_ms": "250", 2024-04-24T14:32:42.778 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_aio_reap_max": "16", 2024-04-24T14:32:42.778 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_async_discard": "false", 2024-04-24T14:32:42.778 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_block_size": "4096", 2024-04-24T14:32:42.778 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_debug_aio": "false", 2024-04-24T14:32:42.778 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-24T14:32:42.778 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-24T14:32:42.778 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_debug_inflight_ios": "false", 2024-04-24T14:32:42.778 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_enable_discard": "false", 2024-04-24T14:32:42.779 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_flock_retry": "3", 2024-04-24T14:32:42.779 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-24T14:32:42.779 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_inject_crash": "0", 2024-04-24T14:32:42.779 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-24T14:32:42.779 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_ioring": "false", 2024-04-24T14:32:42.779 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_ioring_hipri": "false", 2024-04-24T14:32:42.779 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-24T14:32:42.779 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-24T14:32:42.779 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-24T14:32:42.779 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-24T14:32:42.779 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_type": "", 2024-04-24T14:32:42.779 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_alloc_size": "1048576", 2024-04-24T14:32:42.779 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_allocator": "hybrid", 2024-04-24T14:32:42.779 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_buffered_io": "true", 2024-04-24T14:32:42.780 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_check_for_zeros": "false", 2024-04-24T14:32:42.780 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-24T14:32:42.780 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-24T14:32:42.780 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_compact_log_sync": "false", 2024-04-24T14:32:42.780 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-24T14:32:42.780 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-24T14:32:42.780 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-24T14:32:42.780 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-24T14:32:42.780 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_max_log_runway": "4194304", 2024-04-24T14:32:42.780 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_max_prefetch": "1048576", 2024-04-24T14:32:42.780 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_min_flush_size": "524288", 2024-04-24T14:32:42.780 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_min_log_runway": "1048576", 2024-04-24T14:32:42.780 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_replay_recovery": "false", 2024-04-24T14:32:42.781 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-24T14:32:42.781 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-24T14:32:42.781 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_sync_write": "false", 2024-04-24T14:32:42.781 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-24T14:32:42.781 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-24T14:32:42.781 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-24T14:32:42.781 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_allocation_from_file": "true", 2024-04-24T14:32:42.781 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_allocator": "hybrid", 2024-04-24T14:32:42.781 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-24T14:32:42.781 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-24T14:32:42.781 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-24T14:32:42.781 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-24T14:32:42.781 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-24T14:32:42.781 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-24T14:32:42.782 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-24T14:32:42.782 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_create": "true", 2024-04-24T14:32:42.782 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_db_create": "false", 2024-04-24T14:32:42.782 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_db_path": "", 2024-04-24T14:32:42.782 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_db_size": "0", 2024-04-24T14:32:42.782 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_path": "", 2024-04-24T14:32:42.782 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_preallocate_file": "false", 2024-04-24T14:32:42.782 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_size": "107374182400", 2024-04-24T14:32:42.782 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_wal_create": "false", 2024-04-24T14:32:42.782 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_wal_path": "", 2024-04-24T14:32:42.782 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_wal_size": "100663296", 2024-04-24T14:32:42.782 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bluefs": "true", 2024-04-24T14:32:42.782 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-24T14:32:42.783 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-24T14:32:42.783 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-24T14:32:42.783 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-24T14:32:42.783 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-24T14:32:42.783 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-24T14:32:42.783 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-24T14:32:42.783 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-24T14:32:42.783 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_autotune": "true", 2024-04-24T14:32:42.783 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-24T14:32:42.783 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-24T14:32:42.783 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-24T14:32:42.783 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-24T14:32:42.783 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_size": "0", 2024-04-24T14:32:42.784 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-24T14:32:42.784 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-24T14:32:42.784 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-24T14:32:42.784 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-24T14:32:42.784 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_type": "2q", 2024-04-24T14:32:42.784 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-24T14:32:42.784 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_clone_cow": "true", 2024-04-24T14:32:42.784 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-24T14:32:42.784 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-24T14:32:42.784 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-24T14:32:42.784 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-24T14:32:42.784 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-24T14:32:42.784 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-24T14:32:42.784 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-24T14:32:42.785 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_mode": "none", 2024-04-24T14:32:42.785 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-24T14:32:42.785 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_csum_type": "crc32c", 2024-04-24T14:32:42.785 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-24T14:32:42.785 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_freelist": "false", 2024-04-24T14:32:42.785 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-24T14:32:42.785 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-24T14:32:42.785 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_inject_bug21040": "false", 2024-04-24T14:32:42.785 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-24T14:32:42.785 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-24T14:32:42.785 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-24T14:32:42.785 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-24T14:32:42.785 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-24T14:32:42.785 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-24T14:32:42.786 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-24T14:32:42.786 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-24T14:32:42.786 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-24T14:32:42.786 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-24T14:32:42.786 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-24T14:32:42.786 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_small_allocations": "0", 2024-04-24T14:32:42.786 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-24T14:32:42.786 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_default_buffered_read": "true", 2024-04-24T14:32:42.786 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_default_buffered_write": "false", 2024-04-24T14:32:42.786 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-24T14:32:42.786 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-24T14:32:42.786 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-24T14:32:42.786 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-24T14:32:42.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-24T14:32:42.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-24T14:32:42.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-24T14:32:42.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-24T14:32:42.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fail_eio": "false", 2024-04-24T14:32:42.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-24T14:32:42.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-24T14:32:42.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-24T14:32:42.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-24T14:32:42.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-24T14:32:42.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-24T14:32:42.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_mount": "false", 2024-04-24T14:32:42.787 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-24T14:32:42.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_umount": "false", 2024-04-24T14:32:42.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-24T14:32:42.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-24T14:32:42.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-24T14:32:42.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-24T14:32:42.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-24T14:32:42.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-24T14:32:42.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-24T14:32:42.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-24T14:32:42.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_ignore_data_csum": "false", 2024-04-24T14:32:42.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-24T14:32:42.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-24T14:32:42.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-24T14:32:42.788 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_log_op_age": "5.000000", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_alloc_size": "0", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_blob_size": "0", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_deferred_txc": "32", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_min_alloc_size": "0", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_nid_prealloc": "1024", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-24T14:32:42.789 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-24T14:32:42.790 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_retry_disk_reads": "3", 2024-04-24T14:32:42.790 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_rocksdb_cf": "true", 2024-04-24T14:32:42.790 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-04-24T14:32:42.790 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-04-24T14:32:42.790 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-24T14:32:42.790 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-24T14:32:42.790 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-24T14:32:42.790 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-24T14:32:42.790 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_spdk_mem": "512", 2024-04-24T14:32:42.790 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-24T14:32:42.790 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-24T14:32:42.791 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-24T14:32:42.791 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-24T14:32:42.791 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-24T14:32:42.791 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-24T14:32:42.791 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-24T14:32:42.791 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_tracing": "false", 2024-04-24T14:32:42.791 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-24T14:32:42.791 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-24T14:32:42.791 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-24T14:32:42.791 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-24T14:32:42.791 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-24T14:32:42.791 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-24T14:32:42.791 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-24T14:32:42.791 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-24T14:32:42.792 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-24T14:32:42.792 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_zero_block_detection": "false", 2024-04-24T14:32:42.792 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-24T14:32:42.792 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-24T14:32:42.792 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-24T14:32:42.792 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-24T14:32:42.792 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-24T14:32:42.792 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-24T14:32:42.792 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-24T14:32:42.792 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-24T14:32:42.792 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-24T14:32:42.792 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-24T14:32:42.792 INFO:teuthology.orchestra.run.smithi090.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-24T14:32:42.793 INFO:teuthology.orchestra.run.smithi090.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-24T14:32:42.793 INFO:teuthology.orchestra.run.smithi090.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-24T14:32:42.793 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_cluster_require_signatures": "false", 2024-04-24T14:32:42.793 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_cluster_require_version": "2", 2024-04-24T14:32:42.793 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_require_signatures": "false", 2024-04-24T14:32:42.793 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_require_version": "2", 2024-04-24T14:32:42.793 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_service_require_signatures": "false", 2024-04-24T14:32:42.793 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_service_require_version": "2", 2024-04-24T14:32:42.793 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_sign_messages": "true", 2024-04-24T14:32:42.793 INFO:teuthology.orchestra.run.smithi090.stdout: "chdir": "", 2024-04-24T14:32:42.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_acl_type": "", 2024-04-24T14:32:42.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_asio_thread_count": "2", 2024-04-24T14:32:42.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_cache_mid": "0.750000", 2024-04-24T14:32:42.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_cache_size": "16384", 2024-04-24T14:32:42.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_caps_release_delay": "5", 2024-04-24T14:32:42.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_check_pool_perm": "true", 2024-04-24T14:32:42.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-24T14:32:42.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_debug_force_sync_read": "false", 2024-04-24T14:32:42.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_debug_getattr_caps": "false", 2024-04-24T14:32:42.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_debug_inject_tick_delay": "0", 2024-04-24T14:32:42.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-24T14:32:42.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_die_on_failed_remount": "false", 2024-04-24T14:32:42.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_dirsize_rbytes": "true", 2024-04-24T14:32:42.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_force_lazyio": "false", 2024-04-24T14:32:42.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_fs": "", 2024-04-24T14:32:42.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-24T14:32:42.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_inject_release_failure": "false", 2024-04-24T14:32:42.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_max_inline_size": "4096", 2024-04-24T14:32:42.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-24T14:32:42.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mds_namespace": "", 2024-04-24T14:32:42.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_metadata": "", 2024-04-24T14:32:42.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mount_gid": "-1", 2024-04-24T14:32:42.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mount_timeout": "300", 2024-04-24T14:32:42.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mount_uid": "-1", 2024-04-24T14:32:42.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mountpoint": "/", 2024-04-24T14:32:42.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_notify_timeout": "10", 2024-04-24T14:32:42.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc": "true", 2024-04-24T14:32:42.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_max_dirty": "104857600", 2024-04-24T14:32:42.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-24T14:32:42.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_max_objects": "1000", 2024-04-24T14:32:42.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_size": "209715200", 2024-04-24T14:32:42.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_target_dirty": "8388608", 2024-04-24T14:32:42.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_permissions": "true", 2024-04-24T14:32:42.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_quota": "true", 2024-04-24T14:32:42.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_quota_df": "true", 2024-04-24T14:32:42.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_readahead_max_bytes": "0", 2024-04-24T14:32:42.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_readahead_max_periods": "4", 2024-04-24T14:32:42.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_readahead_min": "131072", 2024-04-24T14:32:42.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_reconnect_stale": "false", 2024-04-24T14:32:42.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_shutdown_timeout": "30", 2024-04-24T14:32:42.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_snapdir": ".snap", 2024-04-24T14:32:42.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_tick_interval": "1", 2024-04-24T14:32:42.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_trace": "", 2024-04-24T14:32:42.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_try_dentry_invalidate": "false", 2024-04-24T14:32:42.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client_use_faked_inos": "false", 2024-04-24T14:32:42.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client_use_random_mds": "false", 2024-04-24T14:32:42.797 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_graylog": "false", 2024-04-24T14:32:42.797 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-24T14:32:42.797 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_graylog_port": "12201", 2024-04-24T14:32:42.797 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_monitors": "default=true", 2024-04-24T14:32:42.797 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_syslog": "false", 2024-04-24T14:32:42.797 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-24T14:32:42.797 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_syslog_level": "info", 2024-04-24T14:32:42.797 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "-", 2024-04-24T14:32:42.797 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_network": "", 2024-04-24T14:32:42.797 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_network_interface": "", 2024-04-24T14:32:42.797 INFO:teuthology.orchestra.run.smithi090.stdout: "compressor_zlib_isal": "false", 2024-04-24T14:32:42.798 INFO:teuthology.orchestra.run.smithi090.stdout: "compressor_zlib_level": "5", 2024-04-24T14:32:42.798 INFO:teuthology.orchestra.run.smithi090.stdout: "compressor_zlib_winsize": "-15", 2024-04-24T14:32:42.798 INFO:teuthology.orchestra.run.smithi090.stdout: "compressor_zstd_level": "1", 2024-04-24T14:32:42.798 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-24T14:32:42.798 INFO:teuthology.orchestra.run.smithi090.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-24T14:32:42.798 INFO:teuthology.orchestra.run.smithi090.stdout: "crimson_alien_op_num_threads": "6", 2024-04-24T14:32:42.798 INFO:teuthology.orchestra.run.smithi090.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-24T14:32:42.798 INFO:teuthology.orchestra.run.smithi090.stdout: "crimson_osd_obc_lru_size": "10", 2024-04-24T14:32:42.798 INFO:teuthology.orchestra.run.smithi090.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-24T14:32:42.798 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_location": "", 2024-04-24T14:32:42.798 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_location_hook": "", 2024-04-24T14:32:42.798 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_location_hook_timeout": "10", 2024-04-24T14:32:42.798 INFO:teuthology.orchestra.run.smithi090.stdout: "daemonize": "false", 2024-04-24T14:32:42.799 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_alienstore": "0/5", 2024-04-24T14:32:42.799 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_allow_any_pool_priority": "false", 2024-04-24T14:32:42.799 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_asok": "1/5", 2024-04-24T14:32:42.799 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_asok_assert_abort": "false", 2024-04-24T14:32:42.799 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_asserts_on_shutdown": "true", 2024-04-24T14:32:42.799 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_auth": "20/20", 2024-04-24T14:32:42.799 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_bdev": "1/3", 2024-04-24T14:32:42.799 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_bluefs": "1/5", 2024-04-24T14:32:42.799 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_bluestore": "1/5", 2024-04-24T14:32:42.799 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_buffer": "0/1", 2024-04-24T14:32:42.799 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_ceph_exporter": "1/5", 2024-04-24T14:32:42.799 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_cephfs_mirror": "0/5", 2024-04-24T14:32:42.799 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_cephsqlite": "0/5", 2024-04-24T14:32:42.799 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_civetweb": "1/10", 2024-04-24T14:32:42.800 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_client": "0/5", 2024-04-24T14:32:42.800 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_compressor": "1/5", 2024-04-24T14:32:42.800 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_context": "0/1", 2024-04-24T14:32:42.800 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_crush": "1/1", 2024-04-24T14:32:42.800 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_crypto": "1/5", 2024-04-24T14:32:42.800 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_deliberately_leak_memory": "false", 2024-04-24T14:32:42.800 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_disable_randomized_ping": "false", 2024-04-24T14:32:42.800 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_dpdk": "1/5", 2024-04-24T14:32:42.800 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_eventtrace": "1/5", 2024-04-24T14:32:42.800 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_filer": "0/1", 2024-04-24T14:32:42.800 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_filestore": "1/3", 2024-04-24T14:32:42.800 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_finisher": "1/1", 2024-04-24T14:32:42.800 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_fuse": "1/5", 2024-04-24T14:32:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_heartbeat_testing_span": "0", 2024-04-24T14:32:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_heartbeatmap": "1/5", 2024-04-24T14:32:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-24T14:32:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_javaclient": "1/5", 2024-04-24T14:32:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_journal": "1/3", 2024-04-24T14:32:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_journaler": "0/5", 2024-04-24T14:32:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_kstore": "1/5", 2024-04-24T14:32:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_leveldb": "4/5", 2024-04-24T14:32:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_lockdep": "0/1", 2024-04-24T14:32:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mclock": "1/5", 2024-04-24T14:32:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds": "1/5", 2024-04-24T14:32:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_balancer": "1/5", 2024-04-24T14:32:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_locker": "1/5", 2024-04-24T14:32:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_log": "1/5", 2024-04-24T14:32:42.802 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_log_expire": "1/5", 2024-04-24T14:32:42.802 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_migrator": "1/5", 2024-04-24T14:32:42.802 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_memdb": "4/5", 2024-04-24T14:32:42.802 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mgr": "2/5", 2024-04-24T14:32:42.802 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mgrc": "1/5", 2024-04-24T14:32:42.802 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mon": "20/20", 2024-04-24T14:32:42.802 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_monc": "0/10", 2024-04-24T14:32:42.802 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_ms": "1/1", 2024-04-24T14:32:42.802 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_none": "0/5", 2024-04-24T14:32:42.802 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_objclass": "0/5", 2024-04-24T14:32:42.802 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_objectcacher": "0/5", 2024-04-24T14:32:42.802 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_objecter": "0/1", 2024-04-24T14:32:42.802 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_optracker": "0/5", 2024-04-24T14:32:42.803 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_osd": "1/5", 2024-04-24T14:32:42.803 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_paxos": "20/20", 2024-04-24T14:32:42.803 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_perfcounter": "1/5", 2024-04-24T14:32:42.803 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_prioritycache": "1/5", 2024-04-24T14:32:42.803 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rados": "0/5", 2024-04-24T14:32:42.803 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rbd": "0/5", 2024-04-24T14:32:42.803 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rbd_mirror": "0/5", 2024-04-24T14:32:42.803 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rbd_pwl": "0/5", 2024-04-24T14:32:42.803 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rbd_replay": "0/5", 2024-04-24T14:32:42.803 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_refs": "0/0", 2024-04-24T14:32:42.803 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_reserver": "1/1", 2024-04-24T14:32:42.803 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw": "1/5", 2024-04-24T14:32:42.803 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw_datacache": "1/5", 2024-04-24T14:32:42.803 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw_sync": "1/5", 2024-04-24T14:32:42.804 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rocksdb": "4/5", 2024-04-24T14:32:42.804 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore": "0/5", 2024-04-24T14:32:42.804 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_cache": "0/5", 2024-04-24T14:32:42.804 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_cleaner": "0/5", 2024-04-24T14:32:42.804 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_device": "0/5", 2024-04-24T14:32:42.804 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_journal": "0/5", 2024-04-24T14:32:42.804 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_lba": "0/5", 2024-04-24T14:32:42.804 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_odata": "0/5", 2024-04-24T14:32:42.804 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_omap": "0/5", 2024-04-24T14:32:42.804 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_onode": "0/5", 2024-04-24T14:32:42.804 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_tm": "0/5", 2024-04-24T14:32:42.804 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_striper": "0/1", 2024-04-24T14:32:42.804 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_test": "0/5", 2024-04-24T14:32:42.805 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_throttle": "1/1", 2024-04-24T14:32:42.805 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_timer": "0/1", 2024-04-24T14:32:42.805 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_tp": "0/5", 2024-04-24T14:32:42.805 INFO:teuthology.orchestra.run.smithi090.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-24T14:32:42.805 INFO:teuthology.orchestra.run.smithi090.stdout: "device_failure_prediction_mode": "none", 2024-04-24T14:32:42.805 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-24T14:32:42.805 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-04-24T14:32:42.805 INFO:teuthology.orchestra.run.smithi090.stdout: "err_to_graylog": "false", 2024-04-24T14:32:42.805 INFO:teuthology.orchestra.run.smithi090.stdout: "err_to_journald": "false", 2024-04-24T14:32:42.805 INFO:teuthology.orchestra.run.smithi090.stdout: "err_to_stderr": "true", 2024-04-24T14:32:42.805 INFO:teuthology.orchestra.run.smithi090.stdout: "err_to_syslog": "false", 2024-04-24T14:32:42.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event_tracing": "false", 2024-04-24T14:32:42.805 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_addr": "0.0.0.0", 2024-04-24T14:32:42.805 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_http_port": "9926", 2024-04-24T14:32:42.806 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_prio_limit": "5", 2024-04-24T14:32:42.806 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-24T14:32:42.806 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_sort_metrics": "true", 2024-04-24T14:32:42.806 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_stats_period": "5", 2024-04-24T14:32:42.806 INFO:teuthology.orchestra.run.smithi090.stdout: "fake_statfs_for_testing": "0", 2024-04-24T14:32:42.806 INFO:teuthology.orchestra.run.smithi090.stdout: "fatal_signal_handlers": "true", 2024-04-24T14:32:42.806 INFO:teuthology.orchestra.run.smithi090.stdout: "filer_max_purge_ops": "10", 2024-04-24T14:32:42.806 INFO:teuthology.orchestra.run.smithi090.stdout: "filer_max_truncate_ops": "128", 2024-04-24T14:32:42.806 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_apply_finisher_threads": "1", 2024-04-24T14:32:42.806 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_blackhole": "false", 2024-04-24T14:32:42.806 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_btrfs_clone_range": "true", 2024-04-24T14:32:42.806 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_btrfs_snap": "true", 2024-04-24T14:32:42.806 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_caller_concurrency": "10", 2024-04-24T14:32:42.806 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_collect_device_partition_information": "true", 2024-04-24T14:32:42.807 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_commit_timeout": "600.000000", 2024-04-24T14:32:42.807 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_debug_inject_read_err": "false", 2024-04-24T14:32:42.807 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_debug_omap_check": "false", 2024-04-24T14:32:42.807 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_debug_random_read_err": "0.000000", 2024-04-24T14:32:42.807 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_debug_verify_split": "false", 2024-04-24T14:32:42.807 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_dump_file": "", 2024-04-24T14:32:42.807 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-24T14:32:42.807 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-24T14:32:42.807 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fadvise": "true", 2024-04-24T14:32:42.807 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fail_eio": "true", 2024-04-24T14:32:42.807 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fd_cache_shards": "16", 2024-04-24T14:32:42.807 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fd_cache_size": "128", 2024-04-24T14:32:42.807 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fiemap": "false", 2024-04-24T14:32:42.808 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fiemap_threshold": "4096", 2024-04-24T14:32:42.808 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-24T14:32:42.808 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-24T14:32:42.808 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_inject_stall": "0", 2024-04-24T14:32:42.808 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_journal_parallel": "false", 2024-04-24T14:32:42.808 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_journal_trailing": "false", 2024-04-24T14:32:42.808 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_journal_writeahead": "false", 2024-04-24T14:32:42.808 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_kill_at": "0", 2024-04-24T14:32:42.808 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-24T14:32:42.808 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-24T14:32:42.808 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-24T14:32:42.808 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-24T14:32:42.808 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-24T14:32:42.809 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattrs": "0", 2024-04-24T14:32:42.809 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-24T14:32:42.809 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-24T14:32:42.809 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-24T14:32:42.809 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-24T14:32:42.809 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_xattr_value_size": "0", 2024-04-24T14:32:42.809 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-24T14:32:42.809 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-24T14:32:42.809 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-24T14:32:42.809 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_merge_threshold": "-10", 2024-04-24T14:32:42.809 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-24T14:32:42.809 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_odsync_write": "false", 2024-04-24T14:32:42.809 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_omap_backend": "rocksdb", 2024-04-24T14:32:42.810 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_omap_backend_path": "", 2024-04-24T14:32:42.810 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-24T14:32:42.810 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-24T14:32:42.810 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-24T14:32:42.810 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_op_thread_timeout": "60", 2024-04-24T14:32:42.810 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_op_threads": "2", 2024-04-24T14:32:42.810 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_punch_hole": "false", 2024-04-24T14:32:42.810 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-24T14:32:42.810 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-24T14:32:42.810 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-24T14:32:42.810 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-24T14:32:42.810 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-24T14:32:42.810 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-24T14:32:42.811 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-24T14:32:42.811 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-24T14:32:42.811 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-24T14:32:42.811 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_ops": "50", 2024-04-24T14:32:42.811 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-24T14:32:42.811 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_seek_data_hole": "false", 2024-04-24T14:32:42.811 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_sloppy_crc": "false", 2024-04-24T14:32:42.811 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-24T14:32:42.811 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_splice": "false", 2024-04-24T14:32:42.811 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_split_multiple": "2", 2024-04-24T14:32:42.811 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_split_rand_factor": "20", 2024-04-24T14:32:42.811 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_update_to": "1000", 2024-04-24T14:32:42.811 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-24T14:32:42.811 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-24T14:32:42.812 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-24T14:32:42.812 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-24T14:32:42.812 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-24T14:32:42.812 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-24T14:32:42.812 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_enable": "true", 2024-04-24T14:32:42.812 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-24T14:32:42.812 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-24T14:32:42.812 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-24T14:32:42.812 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-24T14:32:42.812 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-24T14:32:42.812 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-24T14:32:42.812 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_xfs_extsize": "false", 2024-04-24T14:32:42.812 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_zfs_snap": "false", 2024-04-24T14:32:42.812 INFO:teuthology.orchestra.run.smithi090.stdout: "fio_dir": "/tmp/fio", 2024-04-24T14:32:42.813 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "6706935e-d09a-472b-b839-321319fa9e54", 2024-04-24T14:32:42.813 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_allow_other": "true", 2024-04-24T14:32:42.813 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_atomic_o_trunc": "true", 2024-04-24T14:32:42.813 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_big_writes": "true", 2024-04-24T14:32:42.813 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_debug": "false", 2024-04-24T14:32:42.813 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_default_permissions": "false", 2024-04-24T14:32:42.813 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_disable_pagecache": "false", 2024-04-24T14:32:42.813 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_max_write": "0", 2024-04-24T14:32:42.813 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_multithreaded": "true", 2024-04-24T14:32:42.813 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_require_active_mds": "true", 2024-04-24T14:32:42.813 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_set_user_groups": "true", 2024-04-24T14:32:42.813 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_splice_move": "true", 2024-04-24T14:32:42.813 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_splice_read": "true", 2024-04-24T14:32:42.813 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_splice_write": "true", 2024-04-24T14:32:42.814 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-24T14:32:42.814 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_use_invalidate_cb": "true", 2024-04-24T14:32:42.814 INFO:teuthology.orchestra.run.smithi090.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-24T14:32:42.814 INFO:teuthology.orchestra.run.smithi090.stdout: "gss_target_name": "ceph", 2024-04-24T14:32:42.814 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_file": "", 2024-04-24T14:32:42.814 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_inject_failure": "0", 2024-04-24T14:32:42.814 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_interval": "5", 2024-04-24T14:32:42.814 INFO:teuthology.orchestra.run.smithi090.stdout: "host": "smithi090", 2024-04-24T14:32:42.814 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-24T14:32:42.814 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-24T14:32:42.814 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-24T14:32:42.814 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-24T14:32:42.814 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "inject_early_sigterm": "false", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "jaeger_tracing_enable": "false", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_aio": "true", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_align_min_size": "65536", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_block_align": "true", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_block_size": "4096", 2024-04-24T14:32:42.815 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_dio": "true", 2024-04-24T14:32:42.816 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_discard": "false", 2024-04-24T14:32:42.816 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_force_aio": "false", 2024-04-24T14:32:42.816 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_ignore_corruption": "false", 2024-04-24T14:32:42.816 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_max_write_bytes": "10485760", 2024-04-24T14:32:42.816 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_max_write_entries": "100", 2024-04-24T14:32:42.816 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_replay_from": "0", 2024-04-24T14:32:42.816 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-24T14:32:42.816 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-24T14:32:42.816 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-24T14:32:42.816 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-24T14:32:42.816 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_write_header_frequency": "0", 2024-04-24T14:32:42.816 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_zero_on_create": "false", 2024-04-24T14:32:42.816 INFO:teuthology.orchestra.run.smithi090.stdout: "journaler_prefetch_periods": "10", 2024-04-24T14:32:42.817 INFO:teuthology.orchestra.run.smithi090.stdout: "journaler_prezero_periods": "5", 2024-04-24T14:32:42.817 INFO:teuthology.orchestra.run.smithi090.stdout: "journaler_write_head_interval": "15", 2024-04-24T14:32:42.817 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "", 2024-04-24T14:32:42.817 INFO:teuthology.orchestra.run.smithi090.stdout: "keyfile": "", 2024-04-24T14:32:42.817 INFO:teuthology.orchestra.run.smithi090.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-24T14:32:42.817 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_backend": "rocksdb", 2024-04-24T14:32:42.817 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_default_stripe_size": "65536", 2024-04-24T14:32:42.817 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_fsck_on_mount": "false", 2024-04-24T14:32:42.817 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-24T14:32:42.817 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_max_bytes": "67108864", 2024-04-24T14:32:42.817 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_max_ops": "512", 2024-04-24T14:32:42.817 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_nid_prealloc": "1024", 2024-04-24T14:32:42.817 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_onode_map_size": "1024", 2024-04-24T14:32:42.817 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-24T14:32:42.818 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_sync_submit_transaction": "false", 2024-04-24T14:32:42.818 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_sync_transaction": "false", 2024-04-24T14:32:42.818 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_block_size": "65536", 2024-04-24T14:32:42.818 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_bloom_size": "0", 2024-04-24T14:32:42.818 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_cache_size": "536870912", 2024-04-24T14:32:42.818 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_compact_on_mount": "false", 2024-04-24T14:32:42.818 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_compression": "false", 2024-04-24T14:32:42.818 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_log": "", 2024-04-24T14:32:42.818 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_log_to_ceph_log": "true", 2024-04-24T14:32:42.818 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_max_open_files": "0", 2024-04-24T14:32:42.818 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_paranoid": "false", 2024-04-24T14:32:42.818 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_write_buffer_size": "33554432", 2024-04-24T14:32:42.818 INFO:teuthology.orchestra.run.smithi090.stdout: "librados_thread_count": "2", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "lockdep": "false", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "lockdep_force_backtrace": "false", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "log_coarse_timestamps": "true", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "log_flush_on_exit": "false", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "log_graylog_host": "127.0.0.1", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "log_graylog_port": "12201", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "log_max_new": "1000", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "log_max_recent": "10000", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "log_stderr_prefix": "", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "log_stop_at_utilization": "0.970000", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": "true", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_graylog": "false", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_journald": "false", 2024-04-24T14:32:42.819 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_stderr": "false", 2024-04-24T14:32:42.820 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_syslog": "false", 2024-04-24T14:32:42.820 INFO:teuthology.orchestra.run.smithi090.stdout: "max_rotating_auth_attempts": "10", 2024-04-24T14:32:42.820 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-24T14:32:42.820 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_action_on_write_error": "1", 2024-04-24T14:32:42.820 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_alternate_name_max": "8192", 2024-04-24T14:32:42.820 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_asio_thread_count": "2", 2024-04-24T14:32:42.820 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_export_pin": "true", 2024-04-24T14:32:42.820 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_fragment_dirs": "true", 2024-04-24T14:32:42.820 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-24T14:32:42.820 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_fragment_interval": "5", 2024-04-24T14:32:42.820 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-24T14:32:42.820 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-24T14:32:42.820 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_interval": "10", 2024-04-24T14:32:42.821 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_max": "-1", 2024-04-24T14:32:42.821 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_max_until": "-1", 2024-04-24T14:32:42.821 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_merge_size": "50", 2024-04-24T14:32:42.821 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_midchunk": "0.300000", 2024-04-24T14:32:42.821 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-24T14:32:42.821 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_min_start": "0.200000", 2024-04-24T14:32:42.821 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_minchunk": "0.001000", 2024-04-24T14:32:42.821 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_mode": "0", 2024-04-24T14:32:42.821 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_need_max": "1.200000", 2024-04-24T14:32:42.821 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_need_min": "0.800000", 2024-04-24T14:32:42.821 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-24T14:32:42.821 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-24T14:32:42.821 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_split_bits": "3", 2024-04-24T14:32:42.821 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-24T14:32:42.822 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_split_size": "10000", 2024-04-24T14:32:42.822 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-24T14:32:42.822 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_target_decay": "10.000000", 2024-04-24T14:32:42.822 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-24T14:32:42.822 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_beacon_grace": "15.000000", 2024-04-24T14:32:42.822 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_beacon_interval": "4.000000", 2024-04-24T14:32:42.822 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-24T14:32:42.822 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-24T14:32:42.822 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_mid": "0.700000", 2024-04-24T14:32:42.822 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_release_free_interval": "10", 2024-04-24T14:32:42.822 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_reservation": "0.050000", 2024-04-24T14:32:42.822 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-24T14:32:42.822 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_trim_interval": "1", 2024-04-24T14:32:42.823 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_trim_threshold": "262144", 2024-04-24T14:32:42.823 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-24T14:32:42.823 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-24T14:32:42.823 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-24T14:32:42.823 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_client_prealloc_inos": "1000", 2024-04-24T14:32:42.823 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-24T14:32:42.823 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_connect_bootstrapping": "false", 2024-04-24T14:32:42.823 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_damage_table_max_entries": "10000", 2024-04-24T14:32:42.823 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-24T14:32:42.823 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_debug_auth_pins": "false", 2024-04-24T14:32:42.823 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_debug_frag": "false", 2024-04-24T14:32:42.823 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_debug_scatterstat": "false", 2024-04-24T14:32:42.823 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_debug_subtrees": "false", 2024-04-24T14:32:42.823 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_decay_halflife": "5.000000", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_default_dir_hash": "2", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_defer_session_stale": "true", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_deny_all_reconnect": "false", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dir_keys_per_op": "16384", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dir_max_commit_size": "10", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dir_max_entries": "0", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dump_cache_on_map": "false", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_early_reply": "true", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_enable_op_tracker": "true", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_enforce_unique_name": "true", 2024-04-24T14:32:42.824 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-24T14:32:42.825 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-24T14:32:42.825 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_export_ephemeral_random": "true", 2024-04-24T14:32:42.825 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-24T14:32:42.825 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-24T14:32:42.825 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-24T14:32:42.825 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-24T14:32:42.825 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-24T14:32:42.825 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-24T14:32:42.825 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_health_summarize_threshold": "10", 2024-04-24T14:32:42.825 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-24T14:32:42.825 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-24T14:32:42.825 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_health_dummy": "false", 2024-04-24T14:32:42.825 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-24T14:32:42.826 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_migrator_session_race": "false", 2024-04-24T14:32:42.826 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-24T14:32:42.826 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-24T14:32:42.826 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-24T14:32:42.826 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_join_fs": "", 2024-04-24T14:32:42.826 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_journal_format": "1", 2024-04-24T14:32:42.826 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_create_at": "0", 2024-04-24T14:32:42.826 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_export_at": "0", 2024-04-24T14:32:42.826 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_import_at": "0", 2024-04-24T14:32:42.826 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_journal_at": "0", 2024-04-24T14:32:42.826 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_journal_expire_at": "0", 2024-04-24T14:32:42.826 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_journal_replay_at": "0", 2024-04-24T14:32:42.826 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_link_at": "0", 2024-04-24T14:32:42.827 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_mdstable_at": "0", 2024-04-24T14:32:42.827 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_openc_at": "0", 2024-04-24T14:32:42.827 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_rename_at": "0", 2024-04-24T14:32:42.827 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-24T14:32:42.827 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_events_per_segment": "1024", 2024-04-24T14:32:42.827 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_max_events": "-1", 2024-04-24T14:32:42.827 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_max_segments": "128", 2024-04-24T14:32:42.827 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_pause": "false", 2024-04-24T14:32:42.827 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_segment_size": "0", 2024-04-24T14:32:42.827 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-24T14:32:42.827 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_warn_factor": "2.000000", 2024-04-24T14:32:42.827 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_caps_per_client": "1048576", 2024-04-24T14:32:42.827 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_completed_flushes": "100000", 2024-04-24T14:32:42.827 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_completed_requests": "100000", 2024-04-24T14:32:42.828 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_export_size": "20971520", 2024-04-24T14:32:42.828 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_file_recover": "32", 2024-04-24T14:32:42.828 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_purge_files": "64", 2024-04-24T14:32:42.828 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_purge_ops": "8192", 2024-04-24T14:32:42.828 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-24T14:32:42.828 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-24T14:32:42.828 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_snaps_per_dir": "100", 2024-04-24T14:32:42.828 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_xattr_pairs_size": "65536", 2024-04-24T14:32:42.828 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_metrics_update_interval": "2", 2024-04-24T14:32:42.828 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_min_caps_per_client": "100", 2024-04-24T14:32:42.828 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_min_caps_working_set": "10000", 2024-04-24T14:32:42.828 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-24T14:32:42.828 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_numa_node": "-1", 2024-04-24T14:32:42.828 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-24T14:32:42.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_complaint_time": "30.000000", 2024-04-24T14:32:42.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_history_duration": "600", 2024-04-24T14:32:42.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_history_size": "20", 2024-04-24T14:32:42.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_history_slow_op_size": "20", 2024-04-24T14:32:42.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-24T14:32:42.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_log_threshold": "5", 2024-04-24T14:32:42.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_ping_grace": "15", 2024-04-24T14:32:42.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_ping_interval": "5", 2024-04-24T14:32:42.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-24T14:32:42.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-24T14:32:42.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_max_caps": "30000", 2024-04-24T14:32:42.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-24T14:32:42.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-24T14:32:42.829 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-24T14:32:42.830 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_warning_threshold": "262144", 2024-04-24T14:32:42.830 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-24T14:32:42.830 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_replay_interval": "1.000000", 2024-04-24T14:32:42.830 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-24T14:32:42.830 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-24T14:32:42.830 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_root_ino_gid": "0", 2024-04-24T14:32:42.830 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_root_ino_uid": "0", 2024-04-24T14:32:42.830 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-24T14:32:42.830 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-24T14:32:42.830 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-24T14:32:42.830 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-24T14:32:42.830 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-24T14:32:42.830 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-24T14:32:42.830 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-24T14:32:42.831 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-24T14:32:42.831 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-24T14:32:42.831 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-24T14:32:42.831 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_shutdown_check": "0", 2024-04-24T14:32:42.831 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_skip_ino": "0", 2024-04-24T14:32:42.831 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-24T14:32:42.831 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_snap_max_uid": "4294967294", 2024-04-24T14:32:42.831 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_snap_min_uid": "0", 2024-04-24T14:32:42.831 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_snap_rstat": "false", 2024-04-24T14:32:42.831 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_standby_replay_damaged": "false", 2024-04-24T14:32:42.831 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_symlink_recovery": "true", 2024-04-24T14:32:42.831 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-24T14:32:42.831 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_thrash_exports": "0", 2024-04-24T14:32:42.831 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_thrash_fragments": "0", 2024-04-24T14:32:42.832 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_tick_interval": "5.000000", 2024-04-24T14:32:42.832 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_valgrind_exit": "false", 2024-04-24T14:32:42.832 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_verify_backtrace": "1", 2024-04-24T14:32:42.832 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_verify_scatter": "false", 2024-04-24T14:32:42.832 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-24T14:32:42.832 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_wipe_sessions": "false", 2024-04-24T14:32:42.832 INFO:teuthology.orchestra.run.smithi090.stdout: "mempool_debug": "false", 2024-04-24T14:32:42.832 INFO:teuthology.orchestra.run.smithi090.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-24T14:32:42.832 INFO:teuthology.orchestra.run.smithi090.stdout: "memstore_device_bytes": "1073741824", 2024-04-24T14:32:42.832 INFO:teuthology.orchestra.run.smithi090.stdout: "memstore_page_set": "false", 2024-04-24T14:32:42.832 INFO:teuthology.orchestra.run.smithi090.stdout: "memstore_page_size": "65536", 2024-04-24T14:32:42.832 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_client_bytes": "134217728", 2024-04-24T14:32:42.832 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_client_messages": "512", 2024-04-24T14:32:42.832 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-24T14:32:42.833 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-24T14:32:42.833 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-24T14:32:42.833 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-24T14:32:42.833 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_disabled_modules": "diskprediction_local", 2024-04-24T14:32:42.833 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-24T14:32:42.833 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_max_pg_num_change": "128", 2024-04-24T14:32:42.833 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_mds_bytes": "134217728", 2024-04-24T14:32:42.833 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_mds_messages": "128", 2024-04-24T14:32:42.833 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-24T14:32:42.833 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_mon_bytes": "134217728", 2024-04-24T14:32:42.833 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_mon_messages": "128", 2024-04-24T14:32:42.833 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_osd_bytes": "536870912", 2024-04-24T14:32:42.833 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_osd_messages": "8192", 2024-04-24T14:32:42.833 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_pool": "true", 2024-04-24T14:32:42.834 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-24T14:32:42.834 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_standby_modules": "true", 2024-04-24T14:32:42.834 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_stats_period": "5", 2024-04-24T14:32:42.834 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_stats_threshold": "5", 2024-04-24T14:32:42.834 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_tick_period": "2", 2024-04-24T14:32:42.834 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-24T14:32:42.834 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-24T14:32:42.834 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_allow_pool_delete": "true", 2024-04-24T14:32:42.834 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_allow_pool_size_one": "true", 2024-04-24T14:32:42.834 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_auth_validate_all_caps": "true", 2024-04-24T14:32:42.834 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-24T14:32:42.834 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-24T14:32:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_bytes": "104857600", 2024-04-24T14:32:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_directed_command_retry": "5", 2024-04-24T14:32:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-24T14:32:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-24T14:32:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-24T14:32:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-24T14:32:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_parallel": "3", 2024-04-24T14:32:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_log_interval": "1.000000", 2024-04-24T14:32:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-24T14:32:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_ping_interval": "10.000000", 2024-04-24T14:32:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-24T14:32:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-24T14:32:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-24T14:32:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-24T14:32:42.836 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_file_level": "debug", 2024-04-24T14:32:42.836 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_file": "true", 2024-04-24T14:32:42.836 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-24T14:32:42.836 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-24T14:32:42.836 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-24T14:32:42.836 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_journald": "false", 2024-04-24T14:32:42.836 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-24T14:32:42.836 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-24T14:32:42.836 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-24T14:32:42.836 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-24T14:32:42.836 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_compact_on_bootstrap": "false", 2024-04-24T14:32:42.836 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_compact_on_start": "false", 2024-04-24T14:32:42.836 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_compact_on_trim": "true", 2024-04-24T14:32:42.836 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-24T14:32:42.837 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-24T14:32:42.837 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-24T14:32:42.837 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cpu_threads": "4", 2024-04-24T14:32:42.837 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_crush_min_required_version": "hammer", 2024-04-24T14:32:42.837 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_daemon_bytes": "419430400", 2024-04-24T14:32:42.837 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-24T14:32:42.837 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_data_avail_crit": "5", 2024-04-24T14:32:42.837 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_data_avail_warn": "5", 2024-04-24T14:32:42.837 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_data_size_warn": "16106127360", 2024-04-24T14:32:42.837 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-24T14:32:42.837 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-24T14:32:42.837 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_dump_json": "false", 2024-04-24T14:32:42.837 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-24T14:32:42.838 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_dump_transactions": "false", 2024-04-24T14:32:42.838 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_extra_checks": "false", 2024-04-24T14:32:42.838 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-24T14:32:42.838 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-24T14:32:42.838 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_no_require_pacific": "false", 2024-04-24T14:32:42.838 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_no_require_quincy": "false", 2024-04-24T14:32:42.838 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-24T14:32:42.838 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-24T14:32:42.838 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-24T14:32:42.838 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_down_mkfs_grace": "120", 2024-04-24T14:32:42.838 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_election_default_strategy": "1", 2024-04-24T14:32:42.838 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_election_timeout": "5.000000", 2024-04-24T14:32:42.838 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-24T14:32:42.839 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_elector_ping_divisor": "2", 2024-04-24T14:32:42.839 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-24T14:32:42.839 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_enable_op_tracker": "true", 2024-04-24T14:32:42.839 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_fake_pool_delete": "false", 2024-04-24T14:32:42.839 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_force_quorum_join": "false", 2024-04-24T14:32:42.839 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_globalid_prealloc": "10000", 2024-04-24T14:32:42.839 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_detail_to_clog": "false", 2024-04-24T14:32:42.839 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_log_update_period": "5", 2024-04-24T14:32:42.839 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_max_detail": "50", 2024-04-24T14:32:42.839 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_to_clog": "true", 2024-04-24T14:32:42.839 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_to_clog_interval": "600", 2024-04-24T14:32:42.839 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-24T14:32:42.839 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_host": "172.21.15.90,[v2:172.21.15.90:3301,v1:172.21.15.90:6790],172.21.15.103", 2024-04-24T14:32:42.839 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_host_override": "", 2024-04-24T14:32:42.840 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_initial_members": "", 2024-04-24T14:32:42.840 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-24T14:32:42.840 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-24T14:32:42.840 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-24T14:32:42.840 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-24T14:32:42.840 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-24T14:32:42.840 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_lease": "5.000000", 2024-04-24T14:32:42.840 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-24T14:32:42.840 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-24T14:32:42.840 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_log_full_interval": "50", 2024-04-24T14:32:42.840 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_log_max": "10000", 2024-04-24T14:32:42.840 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_log_max_summary": "50", 2024-04-24T14:32:42.840 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-24T14:32:42.841 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_log_epochs": "500", 2024-04-24T14:32:42.841 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-24T14:32:42.841 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-24T14:32:42.841 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_osd": "10000", 2024-04-24T14:32:42.841 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_pg_per_osd": "10000", 2024-04-24T14:32:42.841 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_pool_pg_num": "65536", 2024-04-24T14:32:42.841 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-24T14:32:42.841 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-24T14:32:42.841 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mds_force_trim_to": "0", 2024-04-24T14:32:42.841 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mds_skip_sanity": "false", 2024-04-24T14:32:42.841 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_memory_autotune": "true", 2024-04-24T14:32:42.841 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_memory_target": "2147483648", 2024-04-24T14:32:42.841 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_beacon_grace": "30", 2024-04-24T14:32:42.841 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-24T14:32:42.842 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_digest_period": "5", 2024-04-24T14:32:42.842 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_inactive_grace": "60", 2024-04-24T14:32:42.842 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-24T14:32:42.842 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-24T14:32:42.842 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_min_osdmap_epochs": "500", 2024-04-24T14:32:42.842 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_complaint_time": "30", 2024-04-24T14:32:42.842 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_history_duration": "600", 2024-04-24T14:32:42.842 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_history_size": "20", 2024-04-24T14:32:42.842 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_history_slow_op_size": "20", 2024-04-24T14:32:42.842 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-24T14:32:42.842 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_log_threshold": "5", 2024-04-24T14:32:42.842 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-24T14:32:42.842 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-24T14:32:42.842 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-24T14:32:42.843 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_auto_mark_in": "false", 2024-04-24T14:32:42.843 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-24T14:32:42.843 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-24T14:32:42.843 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-24T14:32:42.843 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_cache_size": "500", 2024-04-24T14:32:42.843 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-24T14:32:42.843 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-24T14:32:42.843 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-24T14:32:42.843 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_down_out_interval": "600", 2024-04-24T14:32:42.843 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-24T14:32:42.843 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-24T14:32:42.843 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_force_trim_to": "0", 2024-04-24T14:32:42.843 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-24T14:32:42.843 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-24T14:32:42.844 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-24T14:32:42.844 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-24T14:32:42.844 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-24T14:32:42.844 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-24T14:32:42.844 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_max_creating_pgs": "1024", 2024-04-24T14:32:42.844 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-24T14:32:42.844 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_min_down_reporters": "2", 2024-04-24T14:32:42.844 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-24T14:32:42.844 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-24T14:32:42.844 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-24T14:32:42.844 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-24T14:32:42.844 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-24T14:32:42.844 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-24T14:32:42.844 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_report_timeout": "900", 2024-04-24T14:32:42.845 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-24T14:32:42.845 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-24T14:32:42.845 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-24T14:32:42.845 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-24T14:32:42.845 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-24T14:32:42.845 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osdmap_full_prune_interval": "10", 2024-04-24T14:32:42.845 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osdmap_full_prune_min": "10000", 2024-04-24T14:32:42.845 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-04-24T14:32:42.845 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-24T14:32:42.845 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_stuck_threshold": "60", 2024-04-24T14:32:42.845 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-24T14:32:42.845 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-24T14:32:42.845 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-24T14:32:42.846 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-24T14:32:42.846 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-24T14:32:42.846 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-24T14:32:42.846 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_probe_timeout": "2.000000", 2024-04-24T14:32:42.846 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_reweight_max_change": "0.050000", 2024-04-24T14:32:42.846 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_reweight_max_osds": "4", 2024-04-24T14:32:42.846 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-24T14:32:42.846 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-24T14:32:42.846 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-24T14:32:42.846 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-24T14:32:42.846 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-24T14:32:42.846 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_interval": "86400", 2024-04-24T14:32:42.846 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_max_keys": "100", 2024-04-24T14:32:42.846 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_timeout": "300", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_session_timeout": "300", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_smart_report_timeout": "5", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stat_smooth_intervals": "6", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stretch_pool_min_size": "2", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stretch_pool_size": "4", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_debug": "false", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_provider_kill_at": "0", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_requester_kill_at": "0", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_timeout": "60.000000", 2024-04-24T14:32:42.847 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_target_pg_per_osd": "100", 2024-04-24T14:32:42.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_tick_interval": "5", 2024-04-24T14:32:42.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_timecheck_interval": "300.000000", 2024-04-24T14:32:42.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-24T14:32:42.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_older_version_delay": "604800", 2024-04-24T14:32:42.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-24T14:32:42.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-24T14:32:42.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-24T14:32:42.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-24T14:32:42.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-24T14:32:42.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-24T14:32:42.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-24T14:32:42.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_misplaced": "false", 2024-04-24T14:32:42.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-24T14:32:42.848 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_older_version": "true", 2024-04-24T14:32:42.849 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-24T14:32:42.849 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_pool_no_app": "false", 2024-04-24T14:32:42.849 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-24T14:32:42.849 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-24T14:32:42.849 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-24T14:32:42.849 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-24T14:32:42.849 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-24T14:32:42.849 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-24T14:32:42.849 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-24T14:32:42.849 INFO:teuthology.orchestra.run.smithi090.stdout: "monmap": "", 2024-04-24T14:32:42.849 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_op_threads": "3", 2024-04-24T14:32:42.849 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-24T14:32:42.849 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_cm": "false", 2024-04-24T14:32:42.850 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_device_name": "", 2024-04-24T14:32:42.850 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_dscp": "96", 2024-04-24T14:32:42.850 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-24T14:32:42.850 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-24T14:32:42.850 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_local_gid": "", 2024-04-24T14:32:42.850 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-24T14:32:42.850 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_port_num": "1", 2024-04-24T14:32:42.850 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-24T14:32:42.850 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-24T14:32:42.850 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-24T14:32:42.850 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-24T14:32:42.850 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_sl": "3", 2024-04-24T14:32:42.850 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_support_srq": "true", 2024-04-24T14:32:42.851 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_type": "ib", 2024-04-24T14:32:42.851 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_reap_threshold": "5", 2024-04-24T14:32:42.851 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_before_connect": "false", 2024-04-24T14:32:42.851 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_ipv4": "true", 2024-04-24T14:32:42.851 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_ipv6": "false", 2024-04-24T14:32:42.851 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_msgr1": "true", 2024-04-24T14:32:42.851 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_msgr2": "true", 2024-04-24T14:32:42.851 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_port_max": "7568", 2024-04-24T14:32:42.851 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_port_min": "6800", 2024-04-24T14:32:42.851 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-24T14:32:42.851 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_retry_count": "3", 2024-04-24T14:32:42.851 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_retry_delay": "5", 2024-04-24T14:32:42.851 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_client": "false", 2024-04-24T14:32:42.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_mds": "false", 2024-04-24T14:32:42.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_mgr": "false", 2024-04-24T14:32:42.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_mon": "false", 2024-04-24T14:32:42.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_osd": "false", 2024-04-24T14:32:42.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_client_mode": "crc secure", 2024-04-24T14:32:42.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_cluster_mode": "crc secure", 2024-04-24T14:32:42.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_cluster_type": "", 2024-04-24T14:32:42.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_compress_secure": "false", 2024-04-24T14:32:42.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_connection_idle_timeout": "900", 2024-04-24T14:32:42.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_connection_ready_timeout": "10", 2024-04-24T14:32:42.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_crc_data": "true", 2024-04-24T14:32:42.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_crc_header": "true", 2024-04-24T14:32:42.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_bad_msg": "false", 2024-04-24T14:32:42.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_bug": "true", 2024-04-24T14:32:42.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_old_message": "true", 2024-04-24T14:32:42.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_skipped_message": "false", 2024-04-24T14:32:42.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-24T14:32:42.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-24T14:32:42.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_coremask": "0xF", 2024-04-24T14:32:42.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-24T14:32:42.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-24T14:32:42.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_enable_tso": "true", 2024-04-24T14:32:42.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-24T14:32:42.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-24T14:32:42.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_hugepages": "", 2024-04-24T14:32:42.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-24T14:32:42.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-24T14:32:42.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_lro": "true", 2024-04-24T14:32:42.854 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_memory_channel": "4", 2024-04-24T14:32:42.854 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-24T14:32:42.854 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_pmd": "", 2024-04-24T14:32:42.854 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_port_id": "0", 2024-04-24T14:32:42.854 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-24T14:32:42.854 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-24T14:32:42.854 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dump_on_send": "false", 2024-04-24T14:32:42.854 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_initial_backoff": "0.200000", 2024-04-24T14:32:42.854 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_delay_max": "1.000000", 2024-04-24T14:32:42.854 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-24T14:32:42.854 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_delay_type": "", 2024-04-24T14:32:42.854 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-24T14:32:42.854 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_socket_failures": "5000", 2024-04-24T14:32:42.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_learn_addr_from_peer": "true", 2024-04-24T14:32:42.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_max_accept_failures": "4", 2024-04-24T14:32:42.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_max_backoff": "15.000000", 2024-04-24T14:32:42.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_mon_client_mode": "secure crc", 2024-04-24T14:32:42.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-24T14:32:42.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_mon_service_mode": "secure crc", 2024-04-24T14:32:42.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_osd_compress_min_size": "1024", 2024-04-24T14:32:42.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_osd_compress_mode": "none", 2024-04-24T14:32:42.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_osd_compression_algorithm": "snappy zlib zstd lz4", 2024-04-24T14:32:42.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-24T14:32:42.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_pq_min_cost": "65536", 2024-04-24T14:32:42.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_public_type": "", 2024-04-24T14:32:42.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_service_mode": "crc secure", 2024-04-24T14:32:42.855 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_tcp_listen_backlog": "512", 2024-04-24T14:32:42.856 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_tcp_nodelay": "true", 2024-04-24T14:32:42.856 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-24T14:32:42.856 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_tcp_rcvbuf": "0", 2024-04-24T14:32:42.856 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_type": "async+posix", 2024-04-24T14:32:42.856 INFO:teuthology.orchestra.run.smithi090.stdout: "no_config_file": "false", 2024-04-24T14:32:42.856 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_completion_locks_per_session": "32", 2024-04-24T14:32:42.856 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-24T14:32:42.856 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-24T14:32:42.856 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_inflight_ops": "1024", 2024-04-24T14:32:42.856 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-24T14:32:42.856 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-24T14:32:42.856 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_tick_interval": "5.000000", 2024-04-24T14:32:42.856 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_timeout": "10.000000", 2024-04-24T14:32:42.856 INFO:teuthology.orchestra.run.smithi090.stdout: "objectstore_blackhole": "false", 2024-04-24T14:32:42.857 INFO:teuthology.orchestra.run.smithi090.stdout: "openssl_engine_opts": "", 2024-04-24T14:32:42.857 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_delay_time": "5.000000", 2024-04-24T14:32:42.857 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_hist_halflife": "1000", 2024-04-24T14:32:42.857 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_max_low_ops": "2", 2024-04-24T14:32:42.857 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_max_ops": "4", 2024-04-24T14:32:42.857 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-24T14:32:42.857 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-24T14:32:42.857 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_slop": "0.020000", 2024-04-24T14:32:42.857 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-24T14:32:42.857 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-24T14:32:42.857 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_asio_thread_count": "2", 2024-04-24T14:32:42.857 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_async_recovery_min_cost": "100", 2024-04-24T14:32:42.857 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-24T14:32:42.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-24T14:32:42.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backfill_scan_max": "512", 2024-04-24T14:32:42.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backfill_scan_min": "64", 2024-04-24T14:32:42.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backoff_on_degraded": "false", 2024-04-24T14:32:42.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backoff_on_peering": "false", 2024-04-24T14:32:42.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backoff_on_unfound": "true", 2024-04-24T14:32:42.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_beacon_report_interval": "300", 2024-04-24T14:32:42.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bench_duration": "30", 2024-04-24T14:32:42.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-24T14:32:42.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bench_max_block_size": "67108864", 2024-04-24T14:32:42.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-24T14:32:42.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_blkin_trace_all": "false", 2024-04-24T14:32:42.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-24T14:32:42.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-24T14:32:42.859 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_check_for_log_corruption": "false", 2024-04-24T14:32:42.859 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-24T14:32:42.859 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-24T14:32:42.859 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-04-24T14:32:42.859 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-24T14:32:42.859 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_class_update_on_start": "true", 2024-04-24T14:32:42.859 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_client_message_cap": "256", 2024-04-24T14:32:42.859 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_client_message_size_cap": "524288000", 2024-04-24T14:32:42.859 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_client_op_priority": "63", 2024-04-24T14:32:42.859 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_client_watch_timeout": "30", 2024-04-24T14:32:42.859 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_command_max_records": "256", 2024-04-24T14:32:42.859 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_compact_on_start": "false", 2024-04-24T14:32:42.859 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-24T14:32:42.860 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-24T14:32:42.860 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-24T14:32:42.860 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_crush_update_on_start": "true", 2024-04-24T14:32:42.860 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_crush_update_weight_set": "true", 2024-04-24T14:32:42.860 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-24T14:32:42.860 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-24T14:32:42.860 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-24T14:32:42.860 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-24T14:32:42.860 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-24T14:32:42.860 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_feed_pullee": "-1", 2024-04-24T14:32:42.860 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-24T14:32:42.860 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-24T14:32:42.860 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-24T14:32:42.860 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_misdirected_ops": "false", 2024-04-24T14:32:42.861 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_no_acting_change": "false", 2024-04-24T14:32:42.861 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_no_purge_strays": "false", 2024-04-24T14:32:42.861 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_op_order": "false", 2024-04-24T14:32:42.861 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-24T14:32:42.861 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-24T14:32:42.861 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-24T14:32:42.861 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-24T14:32:42.861 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_shutdown": "false", 2024-04-24T14:32:42.861 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-24T14:32:42.861 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-24T14:32:42.861 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-24T14:32:42.861 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-24T14:32:42.861 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_verify_snaps": "false", 2024-04-24T14:32:42.861 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-24T14:32:42.862 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-24T14:32:42.862 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_keys": "1024", 2024-04-24T14:32:42.862 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-24T14:32:42.862 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-24T14:32:42.862 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-24T14:32:42.862 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_stride": "524288", 2024-04-24T14:32:42.862 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-24T14:32:42.862 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-24T14:32:42.862 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_default_notify_timeout": "30", 2024-04-24T14:32:42.862 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_delete_sleep": "0.000000", 2024-04-24T14:32:42.862 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-24T14:32:42.862 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-24T14:32:42.862 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_discard_disconnected_ops": "true", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_enable_op_tracker": "true", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_info": "true", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_shutdown": "true", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_function_tracing": "false", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_grace": "20", 2024-04-24T14:32:42.863 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_interval": "6", 2024-04-24T14:32:42.864 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-24T14:32:42.864 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_min_peers": "10", 2024-04-24T14:32:42.864 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_min_size": "2000", 2024-04-24T14:32:42.864 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_stale": "600", 2024-04-24T14:32:42.864 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-24T14:32:42.864 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_hit_set_max_size": "100000", 2024-04-24T14:32:42.864 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_hit_set_min_size": "1000", 2024-04-24T14:32:42.864 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-24T14:32:42.864 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-24T14:32:42.864 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-24T14:32:42.864 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-24T14:32:42.864 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-24T14:32:42.865 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-24T14:32:42.865 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_journal_size": "5120", 2024-04-24T14:32:42.865 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_kill_backfill_at": "0", 2024-04-24T14:32:42.865 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-24T14:32:42.865 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_cache_size": "50", 2024-04-24T14:32:42.865 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_dedup": "true", 2024-04-24T14:32:42.865 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_message_max": "40", 2024-04-24T14:32:42.865 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-24T14:32:42.865 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_share_max_epochs": "40", 2024-04-24T14:32:42.865 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_attr_name_len": "100", 2024-04-24T14:32:42.865 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_attr_size": "0", 2024-04-24T14:32:42.865 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_backfills": "1", 2024-04-24T14:32:42.865 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_markdown_count": "5", 2024-04-24T14:32:42.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_markdown_period": "600", 2024-04-24T14:32:42.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_object_name_len": "2048", 2024-04-24T14:32:42.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_object_namespace_len": "256", 2024-04-24T14:32:42.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_object_size": "134217728", 2024-04-24T14:32:42.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-24T14:32:42.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-24T14:32:42.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_pg_blocked_by": "16", 2024-04-24T14:32:42.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_pg_log_entries": "10000", 2024-04-24T14:32:42.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-24T14:32:42.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_pgls": "1024", 2024-04-24T14:32:42.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_push_cost": "8388608", 2024-04-24T14:32:42.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_push_objects": "10", 2024-04-24T14:32:42.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_scrubs": "3", 2024-04-24T14:32:42.866 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-24T14:32:42.867 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_trimming_pgs": "2", 2024-04-24T14:32:42.867 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_write_op_reply_len": "64", 2024-04-24T14:32:42.867 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_write_size": "90", 2024-04-24T14:32:42.867 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-24T14:32:42.867 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-24T14:32:42.867 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-24T14:32:42.867 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-24T14:32:42.867 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-24T14:32:42.867 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-24T14:32:42.867 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-24T14:32:42.867 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-24T14:32:42.867 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_profile": "balanced", 2024-04-24T14:32:42.867 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-24T14:32:42.867 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-24T14:32:42.868 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-24T14:32:42.868 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-24T14:32:42.868 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-24T14:32:42.868 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-24T14:32:42.868 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-24T14:32:42.868 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-24T14:32:42.868 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-24T14:32:42.868 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-24T14:32:42.868 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-24T14:32:42.868 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_base": "805306368", 2024-04-24T14:32:42.868 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_cache_min": "134217728", 2024-04-24T14:32:42.868 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-24T14:32:42.868 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-24T14:32:42.868 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_target": "4294967296", 2024-04-24T14:32:42.869 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_target_autotune": "false", 2024-04-24T14:32:42.869 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-24T14:32:42.869 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_min_pg_log_entries": "250", 2024-04-24T14:32:42.869 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_min_recovery_priority": "0", 2024-04-24T14:32:42.869 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-24T14:32:42.869 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-24T14:32:42.869 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_report_interval": "5", 2024-04-24T14:32:42.869 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-24T14:32:42.869 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-24T14:32:42.869 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_num_cache_shards": "32", 2024-04-24T14:32:42.869 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_num_op_tracker_shard": "32", 2024-04-24T14:32:42.869 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_numa_auto_affinity": "true", 2024-04-24T14:32:42.869 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_numa_node": "-1", 2024-04-24T14:32:42.869 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_numa_prefer_iface": "true", 2024-04-24T14:32:42.870 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-24T14:32:42.870 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_objecter_finishers": "1", 2024-04-24T14:32:42.870 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_objectstore": "bluestore", 2024-04-24T14:32:42.870 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_objectstore_fuse": "false", 2024-04-24T14:32:42.870 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_objectstore_tracing": "false", 2024-04-24T14:32:42.870 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_complaint_time": "30.000000", 2024-04-24T14:32:42.870 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_history_duration": "600", 2024-04-24T14:32:42.870 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_history_size": "20", 2024-04-24T14:32:42.870 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_history_slow_op_size": "20", 2024-04-24T14:32:42.870 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-24T14:32:42.870 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_log_threshold": "5", 2024-04-24T14:32:42.870 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_shards": "0", 2024-04-24T14:32:42.870 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_shards_hdd": "5", 2024-04-24T14:32:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_shards_ssd": "8", 2024-04-24T14:32:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-24T14:32:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-24T14:32:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-24T14:32:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-24T14:32:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_pq_min_cost": "65536", 2024-04-24T14:32:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-24T14:32:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_queue_cut_off": "high", 2024-04-24T14:32:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-24T14:32:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_thread_timeout": "15", 2024-04-24T14:32:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_open_classes_on_start": "true", 2024-04-24T14:32:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_os_flags": "0", 2024-04-24T14:32:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_peering_op_priority": "255", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_delete_cost": "1048576", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_delete_priority": "5", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_log_trim_max": "10000", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_log_trim_min": "100", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_stat_report_interval_max": "500", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-24T14:32:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-24T14:32:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-24T14:32:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-24T14:32:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-24T14:32:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-24T14:32:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-24T14:32:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-24T14:32:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-24T14:32:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-24T14:32:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-24T14:32:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flags": "0", 2024-04-24T14:32:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-24T14:32:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_min_size": "0", 2024-04-24T14:32:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-24T14:32:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_pg_num": "32", 2024-04-24T14:32:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_pgp_num": "0", 2024-04-24T14:32:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-24T14:32:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_size": "2", 2024-04-24T14:32:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_type": "replicated", 2024-04-24T14:32:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-24T14:32:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-24T14:32:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_push_per_object_cost": "1000", 2024-04-24T14:32:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-24T14:32:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recover_clone_overlap": "true", 2024-04-24T14:32:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-24T14:32:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_cost": "20971520", 2024-04-24T14:32:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-24T14:32:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_active": "0", 2024-04-24T14:32:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-24T14:32:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-24T14:32:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-24T14:32:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-24T14:32:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_single_start": "1", 2024-04-24T14:32:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_op_priority": "3", 2024-04-24T14:32:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-24T14:32:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_priority": "5", 2024-04-24T14:32:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-24T14:32:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_sleep": "0.000000", 2024-04-24T14:32:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-24T14:32:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-24T14:32:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-24T14:32:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_repair_during_recovery": "false", 2024-04-24T14:32:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_requested_scrub_priority": "120", 2024-04-24T14:32:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-24T14:32:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-24T14:32:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_auto_repair": "false", 2024-04-24T14:32:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-24T14:32:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-24T14:32:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_begin_hour": "0", 2024-04-24T14:32:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_begin_week_day": "0", 2024-04-24T14:32:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_chunk_max": "25", 2024-04-24T14:32:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_chunk_min": "5", 2024-04-24T14:32:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_cost": "52428800", 2024-04-24T14:32:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_during_recovery": "false", 2024-04-24T14:32:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_end_hour": "0", 2024-04-24T14:32:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_end_week_day": "0", 2024-04-24T14:32:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_event_cost": "4096", 2024-04-24T14:32:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-24T14:32:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-24T14:32:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_invalid_stats": "true", 2024-04-24T14:32:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-24T14:32:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-24T14:32:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_max_preemptions": "5", 2024-04-24T14:32:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-24T14:32:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_priority": "5", 2024-04-24T14:32:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_sleep": "0.000000", 2024-04-24T14:32:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-24T14:32:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_skip_data_digest": "false", 2024-04-24T14:32:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_smart_report_timeout": "5", 2024-04-24T14:32:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_cost": "1048576", 2024-04-24T14:32:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_priority": "5", 2024-04-24T14:32:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-24T14:32:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-24T14:32:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-24T14:32:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-24T14:32:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-24T14:32:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_target_transaction_size": "30", 2024-04-24T14:32:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-24T14:32:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-24T14:32:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-24T14:32:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-24T14:32:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-24T14:32:42.879 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-24T14:32:42.879 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-24T14:32:42.879 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-24T14:32:42.879 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-24T14:32:42.879 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-24T14:32:42.879 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tracing": "false", 2024-04-24T14:32:42.879 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_use_stale_snap": "false", 2024-04-24T14:32:42.879 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-24T14:32:42.879 INFO:teuthology.orchestra.run.smithi090.stdout: "osdc_blkin_trace_all": "false", 2024-04-24T14:32:42.879 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_kill_at": "0", 2024-04-24T14:32:42.879 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_max_join_drift": "10", 2024-04-24T14:32:42.879 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_min": "500", 2024-04-24T14:32:42.879 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_min_wait": "0.050000", 2024-04-24T14:32:42.880 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_propose_interval": "1.000000", 2024-04-24T14:32:42.880 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_service_trim_max": "500", 2024-04-24T14:32:42.880 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-24T14:32:42.880 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_service_trim_min": "250", 2024-04-24T14:32:42.880 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_stash_full_interval": "25", 2024-04-24T14:32:42.880 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_trim_max": "500", 2024-04-24T14:32:42.880 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_trim_min": "250", 2024-04-24T14:32:42.880 INFO:teuthology.orchestra.run.smithi090.stdout: "perf": "true", 2024-04-24T14:32:42.880 INFO:teuthology.orchestra.run.smithi090.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-24T14:32:42.880 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-24T14:32:42.880 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin_dir": "/usr/lib/ceph", 2024-04-24T14:32:42.880 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "-", 2024-04-24T14:32:42.880 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrv": "", 2024-04-24T14:32:42.881 INFO:teuthology.orchestra.run.smithi090.stdout: "public_bind_addr": "-", 2024-04-24T14:32:42.881 INFO:teuthology.orchestra.run.smithi090.stdout: "public_network": "", 2024-04-24T14:32:42.881 INFO:teuthology.orchestra.run.smithi090.stdout: "public_network_interface": "", 2024-04-24T14:32:42.881 INFO:teuthology.orchestra.run.smithi090.stdout: "qat_compressor_enabled": "false", 2024-04-24T14:32:42.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rados_mon_op_timeout": "0", 2024-04-24T14:32:42.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rados_osd_op_timeout": "0", 2024-04-24T14:32:42.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rados_tracing": "false", 2024-04-24T14:32:42.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_atime_update_interval": "60", 2024-04-24T14:32:42.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-24T14:32:42.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_balance_parent_reads": "false", 2024-04-24T14:32:42.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_balance_snap_reads": "false", 2024-04-24T14:32:42.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_blkin_trace_all": "false", 2024-04-24T14:32:42.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-24T14:32:42.881 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-24T14:32:42.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache": "true", 2024-04-24T14:32:42.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-24T14:32:42.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-24T14:32:42.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-24T14:32:42.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-24T14:32:42.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_policy": "writearound", 2024-04-24T14:32:42.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_size": "33554432", 2024-04-24T14:32:42.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-24T14:32:42.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-24T14:32:42.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_clone_copy_on_read": "false", 2024-04-24T14:32:42.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_compression_hint": "none", 2024-04-24T14:32:42.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_concurrent_management_ops": "10", 2024-04-24T14:32:42.882 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-24T14:32:42.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_clone_format": "auto", 2024-04-24T14:32:42.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_data_pool": "", 2024-04-24T14:32:42.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_features": "61", 2024-04-24T14:32:42.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_format": "2", 2024-04-24T14:32:42.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_map_options": "", 2024-04-24T14:32:42.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_order": "22", 2024-04-24T14:32:42.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_pool": "rbd", 2024-04-24T14:32:42.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-24T14:32:42.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_stripe_count": "0", 2024-04-24T14:32:42.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_stripe_unit": "0", 2024-04-24T14:32:42.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-24T14:32:42.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-24T14:32:42.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-24T14:32:42.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_enable_alloc_hint": "true", 2024-04-24T14:32:42.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-24T14:32:42.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_io_scheduler": "simple", 2024-04-24T14:32:42.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-24T14:32:42.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-24T14:32:42.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-24T14:32:42.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-24T14:32:42.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-24T14:32:42.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-24T14:32:42.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-24T14:32:42.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-24T14:32:42.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-24T14:32:42.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_order": "24", 2024-04-24T14:32:42.884 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_pool": "", 2024-04-24T14:32:42.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_splay_width": "4", 2024-04-24T14:32:42.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_localize_parent_reads": "false", 2024-04-24T14:32:42.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_localize_snap_reads": "false", 2024-04-24T14:32:42.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-24T14:32:42.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-24T14:32:42.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-24T14:32:42.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-24T14:32:42.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-24T14:32:42.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-24T14:32:42.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-24T14:32:42.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-24T14:32:42.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-24T14:32:42.885 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-24T14:32:42.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-24T14:32:42.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-24T14:32:42.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-24T14:32:42.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-24T14:32:42.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-24T14:32:42.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-24T14:32:42.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-24T14:32:42.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-24T14:32:42.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-24T14:32:42.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-24T14:32:42.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-24T14:32:42.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-24T14:32:42.886 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-24T14:32:42.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-24T14:32:42.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-24T14:32:42.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-24T14:32:42.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-24T14:32:42.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-24T14:32:42.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-24T14:32:42.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-24T14:32:42.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-24T14:32:42.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mtime_update_interval": "60", 2024-04-24T14:32:42.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_non_blocking_aio": "true", 2024-04-24T14:32:42.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_op_thread_timeout": "60", 2024-04-24T14:32:42.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_op_threads": "1", 2024-04-24T14:32:42.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_parent_cache_enabled": "false", 2024-04-24T14:32:42.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-24T14:32:42.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-24T14:32:42.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-24T14:32:42.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_plugins": "", 2024-04-24T14:32:42.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_bps_burst": "0", 2024-04-24T14:32:42.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-24T14:32:42.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_bps_limit": "0", 2024-04-24T14:32:42.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_exclude_ops": "0", 2024-04-24T14:32:42.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_iops_burst": "0", 2024-04-24T14:32:42.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-24T14:32:42.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_iops_limit": "0", 2024-04-24T14:32:42.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-24T14:32:42.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-24T14:32:42.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-24T14:32:42.888 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-24T14:32:42.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-24T14:32:42.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-24T14:32:42.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-24T14:32:42.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-24T14:32:42.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-24T14:32:42.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-24T14:32:42.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-24T14:32:42.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-24T14:32:42.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-24T14:32:42.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-24T14:32:42.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_read_from_replica_policy": "default", 2024-04-24T14:32:42.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-24T14:32:42.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-24T14:32:42.889 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-24T14:32:42.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-24T14:32:42.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_skip_partial_discard": "true", 2024-04-24T14:32:42.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-24T14:32:42.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_tracing": "false", 2024-04-24T14:32:42.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_validate_names": "true", 2024-04-24T14:32:42.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_validate_pool": "true", 2024-04-24T14:32:42.890 INFO:teuthology.orchestra.run.smithi090.stdout: "restapi_base_url": "", 2024-04-24T14:32:42.890 INFO:teuthology.orchestra.run.smithi090.stdout: "restapi_log_level": "", 2024-04-24T14:32:42.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_acl_grants_max_num": "100", 2024-04-24T14:32:42.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_admin_entry": "admin", 2024-04-24T14:32:42.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_backend_store": "rados", 2024-04-24T14:32:42.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_barbican_url": "", 2024-04-24T14:32:42.890 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_beast_enable_async": "true", 2024-04-24T14:32:42.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-24T14:32:42.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-24T14:32:42.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-24T14:32:42.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-24T14:32:42.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-24T14:32:42.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cache_enabled": "true", 2024-04-24T14:32:42.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cache_expiry_interval": "900", 2024-04-24T14:32:42.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cache_lru_size": "10000", 2024-04-24T14:32:42.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_content_length_compat": "false", 2024-04-24T14:32:42.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_copy_obj_progress": "true", 2024-04-24T14:32:42.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-24T14:32:42.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cors_rules_max_num": "100", 2024-04-24T14:32:42.891 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cross_domain_policy": "", 2024-04-24T14:32:42.892 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-24T14:32:42.892 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_addr": "", 2024-04-24T14:32:42.892 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-24T14:32:42.892 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-24T14:32:42.892 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-24T14:32:42.892 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-24T14:32:42.892 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_password": "", 2024-04-24T14:32:42.892 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-24T14:32:42.892 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_username": "", 2024-04-24T14:32:42.892 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_require_ssl": "true", 2024-04-24T14:32:42.892 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-24T14:32:42.892 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-24T14:32:42.892 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-24T14:32:42.892 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-24T14:32:42.893 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-24T14:32:42.893 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-24T14:32:42.893 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-24T14:32:42.893 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-24T14:32:42.893 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-24T14:32:42.893 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-24T14:32:42.893 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-24T14:32:42.893 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-24T14:32:42.893 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-24T14:32:42.893 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-24T14:32:42.893 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-24T14:32:42.893 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_addr": "", 2024-04-24T14:32:42.893 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_auth": "token", 2024-04-24T14:32:42.893 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_namespace": "", 2024-04-24T14:32:42.894 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_prefix": "", 2024-04-24T14:32:42.894 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-24T14:32:42.894 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-24T14:32:42.894 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-24T14:32:42.894 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-24T14:32:42.894 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_token_file": "", 2024-04-24T14:32:42.894 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-24T14:32:42.894 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_curl_buffersize": "524288", 2024-04-24T14:32:42.894 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-24T14:32:42.894 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_curl_low_speed_time": "300", 2024-04-24T14:32:42.894 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-24T14:32:42.894 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-24T14:32:42.894 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-24T14:32:42.895 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-24T14:32:42.895 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-24T14:32:42.895 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-24T14:32:42.895 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-24T14:32:42.895 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-24T14:32:42.895 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-24T14:32:42.895 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-24T14:32:42.895 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_log_changes_size": "1000", 2024-04-24T14:32:42.895 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_log_num_shards": "128", 2024-04-24T14:32:42.895 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-24T14:32:42.895 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_log_window": "30", 2024-04-24T14:32:42.895 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_notify_interval_msec": "200", 2024-04-24T14:32:42.895 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-24T14:32:42.896 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-24T14:32:42.896 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-24T14:32:42.896 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-24T14:32:42.896 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-24T14:32:42.896 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-24T14:32:42.896 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-24T14:32:42.896 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-24T14:32:42.896 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-24T14:32:42.896 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-24T14:32:42.896 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-24T14:32:42.896 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-24T14:32:42.896 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-24T14:32:42.896 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-24T14:32:42.896 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-24T14:32:42.897 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-24T14:32:42.897 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-24T14:32:42.897 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-24T14:32:42.897 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-24T14:32:42.897 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-24T14:32:42.897 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-24T14:32:42.897 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dns_name": "", 2024-04-24T14:32:42.897 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dns_s3website_name": "", 2024-04-24T14:32:42.897 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dynamic_resharding": "true", 2024-04-24T14:32:42.897 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-24T14:32:42.897 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_gc_threads": "true", 2024-04-24T14:32:42.897 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_lc_threads": "true", 2024-04-24T14:32:42.897 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_ops_log": "false", 2024-04-24T14:32:42.897 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_quota_threads": "true", 2024-04-24T14:32:42.898 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_static_website": "false", 2024-04-24T14:32:42.898 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_usage_log": "false", 2024-04-24T14:32:42.898 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enforce_swift_acls": "true", 2024-04-24T14:32:42.898 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_exit_timeout_secs": "120", 2024-04-24T14:32:42.898 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_expose_bucket": "false", 2024-04-24T14:32:42.898 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_extended_http_attrs": "", 2024-04-24T14:32:42.898 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-24T14:32:42.898 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_frontends": "beast port=7480", 2024-04-24T14:32:42.898 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-24T14:32:42.898 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_deferred": "50", 2024-04-24T14:32:42.898 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-24T14:32:42.898 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_objs": "32", 2024-04-24T14:32:42.898 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-24T14:32:42.899 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-24T14:32:42.899 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-24T14:32:42.899 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-24T14:32:42.899 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_processor_period": "3600", 2024-04-24T14:32:42.899 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-24T14:32:42.899 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-24T14:32:42.899 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-24T14:32:42.899 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-24T14:32:42.899 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_init_timeout": "300", 2024-04-24T14:32:42.899 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-24T14:32:42.899 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-24T14:32:42.899 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-24T14:32:42.899 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_domain": "", 2024-04-24T14:32:42.900 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_password": "", 2024-04-24T14:32:42.900 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_password_path": "", 2024-04-24T14:32:42.900 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_project": "", 2024-04-24T14:32:42.900 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_tenant": "", 2024-04-24T14:32:42.900 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_token": "", 2024-04-24T14:32:42.900 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_token_path": "", 2024-04-24T14:32:42.900 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_user": "", 2024-04-24T14:32:42.900 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_api_version": "2", 2024-04-24T14:32:42.900 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_domain": "", 2024-04-24T14:32:42.900 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_password": "", 2024-04-24T14:32:42.900 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_project": "", 2024-04-24T14:32:42.900 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-24T14:32:42.900 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_user": "", 2024-04-24T14:32:42.900 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-24T14:32:42.901 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-24T14:32:42.901 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-24T14:32:42.901 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-24T14:32:42.901 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-24T14:32:42.901 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_url": "", 2024-04-24T14:32:42.901 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-24T14:32:42.901 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_debug_interval": "-1", 2024-04-24T14:32:42.901 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_lock_max_time": "90", 2024-04-24T14:32:42.901 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_max_objs": "32", 2024-04-24T14:32:42.901 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_max_rules": "1000", 2024-04-24T14:32:42.901 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_max_worker": "3", 2024-04-24T14:32:42.901 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-24T14:32:42.901 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_thread_delay": "0", 2024-04-24T14:32:42.901 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-24T14:32:42.902 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_dnattr": "uid", 2024-04-24T14:32:42.902 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-24T14:32:42.902 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_searchfilter": "", 2024-04-24T14:32:42.902 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-24T14:32:42.902 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_uri": "", 2024-04-24T14:32:42.902 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-24T14:32:42.902 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-24T14:32:42.902 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-24T14:32:42.902 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_log_http_headers": "", 2024-04-24T14:32:42.902 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-24T14:32:42.902 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-24T14:32:42.902 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_log_object_name_utc": "false", 2024-04-24T14:32:42.902 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-04-24T14:32:42.902 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_attr_name_len": "0", 2024-04-24T14:32:42.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_attr_size": "0", 2024-04-24T14:32:42.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-24T14:32:42.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_chunk_size": "4194304", 2024-04-24T14:32:42.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-24T14:32:42.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-24T14:32:42.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_listing_results": "1000", 2024-04-24T14:32:42.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_notify_retries": "10", 2024-04-24T14:32:42.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-24T14:32:42.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_put_param_size": "1048576", 2024-04-24T14:32:42.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_put_size": "5368709120", 2024-04-24T14:32:42.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_slo_entries": "1000", 2024-04-24T14:32:42.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_md_log_max_shards": "64", 2024-04-24T14:32:42.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-24T14:32:42.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-24T14:32:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_mp_lock_max_time": "600", 2024-04-24T14:32:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-24T14:32:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-24T14:32:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-24T14:32:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-24T14:32:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-24T14:32:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-24T14:32:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-24T14:32:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_max_gc": "300", 2024-04-24T14:32:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-24T14:32:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-24T14:32:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-24T14:32:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-24T14:32:42.904 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-24T14:32:42.905 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-24T14:32:42.905 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-24T14:32:42.905 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_num_async_rados_threads": "32", 2024-04-24T14:32:42.905 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_num_control_oids": "8", 2024-04-24T14:32:42.905 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_numa_node": "-1", 2024-04-24T14:32:42.905 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-24T14:32:42.905 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-24T14:32:42.905 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_objexp_chunk_size": "100", 2024-04-24T14:32:42.905 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_objexp_gc_interval": "600", 2024-04-24T14:32:42.905 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-24T14:32:42.905 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-24T14:32:42.905 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-24T14:32:42.905 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_op_thread_timeout": "600", 2024-04-24T14:32:42.905 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_op_tracing": "false", 2024-04-24T14:32:42.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_opa_token": "", 2024-04-24T14:32:42.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_opa_url": "", 2024-04-24T14:32:42.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_opa_verify_ssl": "true", 2024-04-24T14:32:42.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-24T14:32:42.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ops_log_file_path": "", 2024-04-24T14:32:42.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ops_log_rados": "true", 2024-04-24T14:32:42.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ops_log_socket_path": "", 2024-04-24T14:32:42.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-24T14:32:42.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-24T14:32:42.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_period_push_interval": "2.000000", 2024-04-24T14:32:42.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-24T14:32:42.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-24T14:32:42.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_print_continue": "true", 2024-04-24T14:32:42.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-24T14:32:42.907 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-24T14:32:42.907 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-24T14:32:42.907 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-24T14:32:42.907 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-24T14:32:42.907 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_rados_tracing": "false", 2024-04-24T14:32:42.907 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_realm": "", 2024-04-24T14:32:42.907 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_realm_id": "", 2024-04-24T14:32:42.907 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-24T14:32:42.907 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_region": "", 2024-04-24T14:32:42.907 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-24T14:32:42.907 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-24T14:32:42.907 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-24T14:32:42.907 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-24T14:32:42.908 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_request_uri": "", 2024-04-24T14:32:42.908 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_batch_size": "64", 2024-04-24T14:32:42.908 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-24T14:32:42.908 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_max_aio": "128", 2024-04-24T14:32:42.908 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_num_logs": "16", 2024-04-24T14:32:42.908 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_thread_interval": "600", 2024-04-24T14:32:42.908 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_resolve_cname": "false", 2024-04-24T14:32:42.908 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-24T14:32:42.908 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_run_sync_thread": "true", 2024-04-24T14:32:42.908 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-24T14:32:42.908 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-24T14:32:42.908 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-24T14:32:42.908 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_scheduler_type": "throttler", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_script_uri": "", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_service_provider_name": "", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_client_id": "", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_client_secret": "", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_entry": "sts", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_key": "sts", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-24T14:32:42.909 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_min_session_duration": "900", 2024-04-24T14:32:42.910 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_token_introspection_url": "", 2024-04-24T14:32:42.910 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_account_in_url": "false", 2024-04-24T14:32:42.910 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_auth_entry": "auth", 2024-04-24T14:32:42.910 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_auth_url": "", 2024-04-24T14:32:42.910 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_custom_header": "", 2024-04-24T14:32:42.910 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-24T14:32:42.910 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_need_stats": "true", 2024-04-24T14:32:42.910 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_tenant_name": "", 2024-04-24T14:32:42.910 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_token_expiration": "86400", 2024-04-24T14:32:42.910 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_url": "", 2024-04-24T14:32:42.910 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_url_prefix": "swift", 2024-04-24T14:32:42.910 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-24T14:32:42.910 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-24T14:32:42.911 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_lease_period": "120", 2024-04-24T14:32:42.911 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-24T14:32:42.911 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-24T14:32:42.911 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-24T14:32:42.911 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-24T14:32:42.911 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-24T14:32:42.911 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-24T14:32:42.911 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-24T14:32:42.911 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-24T14:32:42.911 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-24T14:32:42.911 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_thread_pool_size": "512", 2024-04-24T14:32:42.911 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_comment": "", 2024-04-24T14:32:42.911 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_createby": "", 2024-04-24T14:32:42.911 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_encoding": "", 2024-04-24T14:32:42.912 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_flag": "false", 2024-04-24T14:32:42.912 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_origin": "", 2024-04-24T14:32:42.912 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-24T14:32:42.912 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_tracker": "", 2024-04-24T14:32:42.912 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_trust_forwarded_https": "false", 2024-04-24T14:32:42.912 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-24T14:32:42.912 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-24T14:32:42.912 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_usage_max_shards": "32", 2024-04-24T14:32:42.912 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_usage_max_user_shards": "1", 2024-04-24T14:32:42.912 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_use_opa_authz": "false", 2024-04-24T14:32:42.912 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-24T14:32:42.912 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-24T14:32:42.912 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_max_buckets": "1000", 2024-04-24T14:32:42.913 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-24T14:32:42.913 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-24T14:32:42.913 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-24T14:32:42.913 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-24T14:32:42.913 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_unique_email": "true", 2024-04-24T14:32:42.913 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_verify_ssl": "true", 2024-04-24T14:32:42.913 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-24T14:32:42.913 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zone": "", 2024-04-24T14:32:42.913 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zone_id": "", 2024-04-24T14:32:42.913 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-24T14:32:42.913 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zonegroup": "", 2024-04-24T14:32:42.913 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zonegroup_id": "", 2024-04-24T14:32:42.913 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-24T14:32:42.913 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_block_size": "4096", 2024-04-24T14:32:42.914 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-24T14:32:42.914 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-24T14:32:42.914 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-24T14:32:42.914 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-24T14:32:42.914 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-24T14:32:42.914 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_size": "536870912", 2024-04-24T14:32:42.914 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-24T14:32:42.914 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-24T14:32:42.914 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-24T14:32:42.914 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-24T14:32:42.914 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-24T14:32:42.914 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-24T14:32:42.914 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-24T14:32:42.914 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-24T14:32:42.915 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_index_type": "binary_search", 2024-04-24T14:32:42.915 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-24T14:32:42.915 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-24T14:32:42.915 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_partition_filters": "false", 2024-04-24T14:32:42.915 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_perf": "false", 2024-04-24T14:32:42.915 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-24T14:32:42.915 INFO:teuthology.orchestra.run.smithi090.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-24T14:32:42.915 INFO:teuthology.orchestra.run.smithi090.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-24T14:32:42.915 INFO:teuthology.orchestra.run.smithi090.stdout: "run_dir": "/var/run/ceph", 2024-04-24T14:32:42.915 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_block_create": "true", 2024-04-24T14:32:42.915 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_cache_lru_size": "67108864", 2024-04-24T14:32:42.915 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_default_max_object_size": "16777216", 2024-04-24T14:32:42.915 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-24T14:32:42.915 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_device_size": "107374182400", 2024-04-24T14:32:42.916 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_init_rewrite_segments_num_per_device": "6", 2024-04-24T14:32:42.916 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_journal_batch_capacity": "16", 2024-04-24T14:32:42.916 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-24T14:32:42.916 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-24T14:32:42.916 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-24T14:32:42.916 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_segment_size": "67108864", 2024-04-24T14:32:42.916 INFO:teuthology.orchestra.run.smithi090.stdout: "setgroup": "", 2024-04-24T14:32:42.916 INFO:teuthology.orchestra.run.smithi090.stdout: "setuser": "", 2024-04-24T14:32:42.916 INFO:teuthology.orchestra.run.smithi090.stdout: "setuser_match_path": "", 2024-04-24T14:32:42.916 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-24T14:32:42.916 INFO:teuthology.orchestra.run.smithi090.stdout: "thp": "false", 2024-04-24T14:32:42.916 INFO:teuthology.orchestra.run.smithi090.stdout: "threadpool_default_timeout": "60", 2024-04-24T14:32:42.916 INFO:teuthology.orchestra.run.smithi090.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-24T14:32:42.916 INFO:teuthology.orchestra.run.smithi090.stdout: "throttler_perf_counter": "true" 2024-04-24T14:32:42.917 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:42.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-24T14:32:43.044 INFO:teuthology.orchestra.run.smithi090.stdout:mon.a: {} 2024-04-24T14:32:43.063 INFO:teuthology.orchestra.run.smithi090.stdout:mon.b: {} 2024-04-24T14:32:43.084 INFO:teuthology.orchestra.run.smithi090.stdout:mon.c: {} 2024-04-24T14:32:43.086 INFO:teuthology.orchestra.run.smithi090.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T14:32:43.086 INFO:teuthology.orchestra.run.smithi090.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T14:32:43.086 INFO:teuthology.orchestra.run.smithi090.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T14:32:43.101 DEBUG:teuthology.orchestra.run.smithi090:> type ceph-objectstore-tool 2024-04-24T14:32:43.105 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-24T14:32:43.105 DEBUG:teuthology.orchestra.run.smithi103:> type ceph-objectstore-tool 2024-04-24T14:32:43.111 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-24T14:32:43.112 INFO:teuthology.run_tasks:Running task workunit... 2024-04-24T14:32:43.125 INFO:tasks.workunit:Pulling workunits from ref 330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T14:32:43.126 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-24T14:32:43.126 DEBUG:teuthology.orchestra.run.smithi173:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-24T14:32:43.126 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-24T14:32:43.126 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T14:32:43.126 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:32:43.127 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-04-24T14:32:43.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 0.05 2024-04-24T14:32:43.127 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-24T14:32:43.127 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-24T14:32:43.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:32:43.128 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-24T14:32:43.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:32:43.128 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-24T14:32:43.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T14:32:43.130 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:43.130+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:43.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T14:32:43.136 INFO:teuthology.orchestra.run.smithi173.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-24T14:32:43.137 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-24T14:32:43.143 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-24T14:32:43.143 DEBUG:teuthology.orchestra.run.smithi173:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-24T14:32:43.192 INFO:tasks.workunit:timeout=3h 2024-04-24T14:32:43.192 INFO:tasks.workunit:cleanup=True 2024-04-24T14:32:43.192 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T14:32:43.200 INFO:tasks.workunit.client.0.smithi173.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-24T14:32:43.229 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:43.230+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:43.259 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:43.259 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:43.259 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:32:43.259 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:43.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:32:43.290 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:32:43.310 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:32:43.327 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:32:43.330 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:43.330+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:43.348 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:32:43.372 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:32:43.395 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:32:43.397 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:32:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:32:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:32:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:32:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:32:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:32:43.407 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:43.407 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:43.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:32:43.429 INFO:teuthology.orchestra.run.smithi090.stdout:moved 0 / 26 (0%) 2024-04-24T14:32:43.429 INFO:teuthology.orchestra.run.smithi090.stdout:avg 4.33333 2024-04-24T14:32:43.429 INFO:teuthology.orchestra.run.smithi090.stdout:stddev 1.49071 -> 1.49071 (expected baseline 1.90029) 2024-04-24T14:32:43.429 INFO:teuthology.orchestra.run.smithi090.stdout:min osd.2 with 3 -> 3 pgs (0.692308 -> 0.692308 * mean) 2024-04-24T14:32:43.429 INFO:teuthology.orchestra.run.smithi090.stdout:max osd.1 with 7 -> 7 pgs (1.61538 -> 1.61538 * mean) 2024-04-24T14:32:43.429 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:43.429 INFO:teuthology.orchestra.run.smithi090.stdout:oload 110 2024-04-24T14:32:43.429 INFO:teuthology.orchestra.run.smithi090.stdout:max_change 0.05 2024-04-24T14:32:43.429 INFO:teuthology.orchestra.run.smithi090.stdout:max_change_osds 4 2024-04-24T14:32:43.430 INFO:teuthology.orchestra.run.smithi090.stdout:average_utilization 0.0032 2024-04-24T14:32:43.430 INFO:teuthology.orchestra.run.smithi090.stdout:overload_utilization 0.0035 2024-04-24T14:32:43.431 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:43.435+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:43.433 INFO:teuthology.orchestra.run.smithi090.stderr:no change 2024-04-24T14:32:43.443 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-04-24T14:32:43.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 1.0 2024-04-24T14:32:43.518 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:43.518 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:43.527 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:32:43.531 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:43.534+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:43.626 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:43.626 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:43.632 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:43.635+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:43.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:32:43.734 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:43.739+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:43.778 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:43.778 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:43.786 INFO:teuthology.orchestra.run.smithi090.stdout:moved 0 / 26 (0%) 2024-04-24T14:32:43.786 INFO:teuthology.orchestra.run.smithi090.stdout:avg 4.33333 2024-04-24T14:32:43.786 INFO:teuthology.orchestra.run.smithi090.stdout:stddev 1.49071 -> 1.49071 (expected baseline 1.90029) 2024-04-24T14:32:43.786 INFO:teuthology.orchestra.run.smithi090.stdout:min osd.2 with 3 -> 3 pgs (0.692308 -> 0.692308 * mean) 2024-04-24T14:32:43.786 INFO:teuthology.orchestra.run.smithi090.stdout:max osd.1 with 7 -> 7 pgs (1.61538 -> 1.61538 * mean) 2024-04-24T14:32:43.786 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:43.786 INFO:teuthology.orchestra.run.smithi090.stdout:oload 110 2024-04-24T14:32:43.786 INFO:teuthology.orchestra.run.smithi090.stdout:max_change 1 2024-04-24T14:32:43.786 INFO:teuthology.orchestra.run.smithi090.stdout:max_change_osds 4 2024-04-24T14:32:43.786 INFO:teuthology.orchestra.run.smithi090.stdout:average_utilization 0.0032 2024-04-24T14:32:43.786 INFO:teuthology.orchestra.run.smithi090.stdout:overload_utilization 0.0035 2024-04-24T14:32:43.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:32:43.789 INFO:teuthology.orchestra.run.smithi090.stderr:no change 2024-04-24T14:32:43.798 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-04-24T14:32:43.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-04-24T14:32:43.835 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:43.839+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:43.884 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:43.884 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:43.893 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:32:43.935 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:43.938+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:43.988 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:43.989 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:43.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:32:44.037 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:44.039+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:44.114 INFO:teuthology.orchestra.run.smithi090.stdout:moved 0 / 26 (0%) 2024-04-24T14:32:44.114 INFO:teuthology.orchestra.run.smithi090.stdout:avg 4.33333 2024-04-24T14:32:44.114 INFO:teuthology.orchestra.run.smithi090.stdout:stddev 1.49071 -> 1.49071 (expected baseline 1.90029) 2024-04-24T14:32:44.114 INFO:teuthology.orchestra.run.smithi090.stdout:min osd.2 with 3 -> 3 pgs (0.692308 -> 0.692308 * mean) 2024-04-24T14:32:44.114 INFO:teuthology.orchestra.run.smithi090.stdout:max osd.1 with 7 -> 7 pgs (1.61538 -> 1.61538 * mean) 2024-04-24T14:32:44.115 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:44.115 INFO:teuthology.orchestra.run.smithi090.stdout:oload 110 2024-04-24T14:32:44.115 INFO:teuthology.orchestra.run.smithi090.stdout:max_change 3 2024-04-24T14:32:44.115 INFO:teuthology.orchestra.run.smithi090.stdout:max_change_osds 4 2024-04-24T14:32:44.115 INFO:teuthology.orchestra.run.smithi090.stdout:average_utilization 0.0032 2024-04-24T14:32:44.115 INFO:teuthology.orchestra.run.smithi090.stdout:overload_utilization 0.0035 2024-04-24T14:32:44.115 INFO:teuthology.orchestra.run.smithi090.stderr:no change 2024-04-24T14:32:44.126 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-04-24T14:32:44.126 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 1.0 2024-04-24T14:32:44.137 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:44.138+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:44.145 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:44.145 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:44.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:32:44.238 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:44.238+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:44.260 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:44.261 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:44.270 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:32:44.339 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:44.343+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:44.376 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:44.377 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:44.386 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:32:44.420 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-24T14:32:44.431 INFO:teuthology.orchestra.run.smithi090.stdout:moved 0 / 26 (0%) 2024-04-24T14:32:44.431 INFO:teuthology.orchestra.run.smithi090.stdout:avg 4.33333 2024-04-24T14:32:44.431 INFO:teuthology.orchestra.run.smithi090.stdout:stddev 1.49071 -> 1.49071 (expected baseline 1.90029) 2024-04-24T14:32:44.431 INFO:teuthology.orchestra.run.smithi090.stdout:min osd.2 with 3 -> 3 pgs (0.692308 -> 0.692308 * mean) 2024-04-24T14:32:44.431 INFO:teuthology.orchestra.run.smithi090.stdout:max osd.1 with 7 -> 7 pgs (1.61538 -> 1.61538 * mean) 2024-04-24T14:32:44.431 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:44.432 INFO:teuthology.orchestra.run.smithi090.stdout:oload 110 2024-04-24T14:32:44.432 INFO:teuthology.orchestra.run.smithi090.stdout:max_change 1 2024-04-24T14:32:44.432 INFO:teuthology.orchestra.run.smithi090.stdout:max_change_osds 4 2024-04-24T14:32:44.432 INFO:teuthology.orchestra.run.smithi090.stdout:average_utilization 0.0032 2024-04-24T14:32:44.432 INFO:teuthology.orchestra.run.smithi090.stdout:overload_utilization 0.0035 2024-04-24T14:32:44.434 INFO:teuthology.orchestra.run.smithi090.stderr:no change 2024-04-24T14:32:44.440 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:44.442+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:44.444 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-04-24T14:32:44.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-04-24T14:32:44.483 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:44.484 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:44.492 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:32:44.542 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:44.543+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:44.587 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:44.587 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:44.596 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:32:44.643 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:44.647+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:44.736 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:44.736 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:44.743 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:44.747+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:44.745 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:32:44.778 INFO:teuthology.orchestra.run.smithi090.stdout:moved 0 / 26 (0%) 2024-04-24T14:32:44.778 INFO:teuthology.orchestra.run.smithi090.stdout:avg 4.33333 2024-04-24T14:32:44.778 INFO:teuthology.orchestra.run.smithi090.stdout:stddev 1.49071 -> 1.49071 (expected baseline 1.90029) 2024-04-24T14:32:44.778 INFO:teuthology.orchestra.run.smithi090.stdout:min osd.2 with 3 -> 3 pgs (0.692308 -> 0.692308 * mean) 2024-04-24T14:32:44.779 INFO:teuthology.orchestra.run.smithi090.stdout:max osd.1 with 7 -> 7 pgs (1.61538 -> 1.61538 * mean) 2024-04-24T14:32:44.779 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:44.779 INFO:teuthology.orchestra.run.smithi090.stdout:oload 110 2024-04-24T14:32:44.779 INFO:teuthology.orchestra.run.smithi090.stdout:max_change 3 2024-04-24T14:32:44.779 INFO:teuthology.orchestra.run.smithi090.stdout:max_change_osds 4 2024-04-24T14:32:44.779 INFO:teuthology.orchestra.run.smithi090.stdout:average_utilization 0.0032 2024-04-24T14:32:44.779 INFO:teuthology.orchestra.run.smithi090.stdout:overload_utilization 0.0035 2024-04-24T14:32:44.782 INFO:teuthology.orchestra.run.smithi090.stderr:no change 2024-04-24T14:32:44.841 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:44.841 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:44.844 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:44.846+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:44.850 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:32:44.945 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:44.947+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:44.948 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:44.948 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:44.957 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:32:45.046 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:45.047+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:45.056 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:45.056 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:45.065 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:32:45.147 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:45.150+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:45.161 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:45.161 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:45.170 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:32:45.248 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:45.251+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:45.278 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:45.278 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:45.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:32:45.349 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:45.351+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:45.396 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:45.396 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:45.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:32:45.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:32:45.450 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:45.451+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:45.505 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:45.505 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:45.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:32:45.541 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:32:45.550 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:45.551+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:45.562 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:32:45.583 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:32:45.616 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:32:45.619 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:32:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:32:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:32:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:36 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'22, mlcod=13'17)", 2024-04-24T14:32:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:32:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.289517495999998, 2024-04-24T14:32:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043791400000000001, 2024-04-24T14:32:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:32:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:32:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331851+0000", 2024-04-24T14:32:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331864+0000", 2024-04-24T14:32:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2767e-05 2024-04-24T14:32:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331881+0000", 2024-04-24T14:32:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6586e-05 2024-04-24T14:32:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331894+0000", 2024-04-24T14:32:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2976000000000001e-05 2024-04-24T14:32:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332161+0000", 2024-04-24T14:32:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026765399999999998 2024-04-24T14:32:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332197+0000", 2024-04-24T14:32:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5565999999999998e-05 2024-04-24T14:32:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336158+0000", 2024-04-24T14:32:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039616820000000002 2024-04-24T14:32:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336232+0000", 2024-04-24T14:32:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3151999999999994e-05 2024-04-24T14:32:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:37 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'23, mlcod=13'18)", 2024-04-24T14:32:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:32:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.288631018, 2024-04-24T14:32:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040417389999999999, 2024-04-24T14:32:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332738+0000", 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332750+0000", 2024-04-24T14:32:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2432000000000001e-05 2024-04-24T14:32:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332768+0000", 2024-04-24T14:32:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7314000000000001e-05 2024-04-24T14:32:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332780+0000", 2024-04-24T14:32:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2716e-05 2024-04-24T14:32:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332925+0000", 2024-04-24T14:32:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144587 2024-04-24T14:32:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332955+0000", 2024-04-24T14:32:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0246000000000001e-05 2024-04-24T14:32:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336698+0000", 2024-04-24T14:32:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037430559999999998 2024-04-24T14:32:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336781+0000", 2024-04-24T14:32:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2447999999999994e-05 2024-04-24T14:32:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:39 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'25, mlcod=13'22)", 2024-04-24T14:32:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:32:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.282512695000001, 2024-04-24T14:32:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038736740000000001, 2024-04-24T14:32:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:32:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:32:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338856+0000", 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338995+0000", 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013868100000000001 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339029+0000", 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4372000000000002e-05 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339050+0000", 2024-04-24T14:32:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0656999999999999e-05 2024-04-24T14:32:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339156+0000", 2024-04-24T14:32:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106177 2024-04-24T14:32:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339188+0000", 2024-04-24T14:32:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.243e-05 2024-04-24T14:32:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342648+0000", 2024-04-24T14:32:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034601689999999999 2024-04-24T14:32:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342731+0000", 2024-04-24T14:32:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2534999999999995e-05 2024-04-24T14:32:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:67 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'41, mlcod=13'39)", 2024-04-24T14:32:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:32:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.231776410999998, 2024-04-24T14:32:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031029009999999999, 2024-04-24T14:32:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:32:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:32:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389592+0000", 2024-04-24T14:32:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389603+0000", 2024-04-24T14:32:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090000000000001e-05 2024-04-24T14:32:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389631+0000", 2024-04-24T14:32:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7772000000000001e-05 2024-04-24T14:32:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389646+0000", 2024-04-24T14:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.432e-05 2024-04-24T14:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389732+0000", 2024-04-24T14:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6331999999999999e-05 2024-04-24T14:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389771+0000", 2024-04-24T14:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8751999999999998e-05 2024-04-24T14:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392585+0000", 2024-04-24T14:32:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028147469999999998 2024-04-24T14:32:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392696+0000", 2024-04-24T14:32:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111121 2024-04-24T14:32:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T14:32:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:32:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.2306338, 2024-04-24T14:32:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030475519999999998, 2024-04-24T14:32:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:32:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390735+0000", 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390804+0000", 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9488999999999998e-05 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390832+0000", 2024-04-24T14:32:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7946e-05 2024-04-24T14:32:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390847+0000", 2024-04-24T14:32:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4693e-05 2024-04-24T14:32:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391508+0000", 2024-04-24T14:32:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066154799999999998 2024-04-24T14:32:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391538+0000", 2024-04-24T14:32:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0029e-05 2024-04-24T14:32:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393698+0000", 2024-04-24T14:32:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002160052 2024-04-24T14:32:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393784+0000", 2024-04-24T14:32:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5414999999999995e-05 2024-04-24T14:32:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:72 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'46, mlcod=13'43)", 2024-04-24T14:32:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:32:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.224460307000001, 2024-04-24T14:32:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036059460000000001, 2024-04-24T14:32:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396908+0000", 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396968+0000", 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9412e-05 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397010+0000", 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2169000000000002e-05 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397022+0000", 2024-04-24T14:32:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1749999999999999e-05 2024-04-24T14:32:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397096+0000", 2024-04-24T14:32:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4701000000000002e-05 2024-04-24T14:32:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397126+0000", 2024-04-24T14:32:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2024-04-24T14:32:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400443+0000", 2024-04-24T14:32:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033172140000000002 2024-04-24T14:32:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400516+0000", 2024-04-24T14:32:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2181999999999994e-05 2024-04-24T14:32:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:73 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'47, mlcod=13'43)", 2024-04-24T14:32:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:32:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.223612902999999, 2024-04-24T14:32:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055152170000000002, 2024-04-24T14:32:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:32:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:32:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397756+0000", 2024-04-24T14:32:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397884+0000", 2024-04-24T14:32:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012842100000000001 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397923+0000", 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8942000000000001e-05 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397939+0000", 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6073e-05 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398106+0000", 2024-04-24T14:32:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016737100000000001 2024-04-24T14:32:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398137+0000", 2024-04-24T14:32:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0496000000000001e-05 2024-04-24T14:32:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403191+0000", 2024-04-24T14:32:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00505391 2024-04-24T14:32:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403272+0000", 2024-04-24T14:32:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1483999999999994e-05 2024-04-24T14:32:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:74 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'48, mlcod=13'43)", 2024-04-24T14:32:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:32:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.222813877, 2024-04-24T14:32:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048091760000000001, 2024-04-24T14:32:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:32:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:32:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398555+0000", 2024-04-24T14:32:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398570+0000", 2024-04-24T14:32:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.558e-05 2024-04-24T14:32:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398610+0000", 2024-04-24T14:32:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0028999999999999e-05 2024-04-24T14:32:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398625+0000", 2024-04-24T14:32:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-04-24T14:32:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398701+0000", 2024-04-24T14:32:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6507000000000005e-05 2024-04-24T14:32:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398731+0000", 2024-04-24T14:32:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9356000000000002e-05 2024-04-24T14:32:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403281+0000", 2024-04-24T14:32:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004549796 2024-04-24T14:32:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403365+0000", 2024-04-24T14:32:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4685000000000007e-05 2024-04-24T14:32:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-04-24T14:32:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:32:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.201500952, 2024-04-24T14:32:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038140460000000002, 2024-04-24T14:32:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419868+0000", 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419882+0000", 2024-04-24T14:32:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.45e-05 2024-04-24T14:32:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419919+0000", 2024-04-24T14:32:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6939e-05 2024-04-24T14:32:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419961+0000", 2024-04-24T14:32:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2148999999999999e-05 2024-04-24T14:32:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420084+0000", 2024-04-24T14:32:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122958 2024-04-24T14:32:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420122+0000", 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8334999999999998e-05 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423616+0000", 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003493566 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423683+0000", 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7149000000000004e-05 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T14:32:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:32:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.200089197, 2024-04-24T14:32:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030290730000000002, 2024-04-24T14:32:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:32:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421279+0000", 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421352+0000", 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2251999999999999e-05 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421378+0000", 2024-04-24T14:32:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6006000000000001e-05 2024-04-24T14:32:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421388+0000", 2024-04-24T14:32:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0373e-05 2024-04-24T14:32:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421473+0000", 2024-04-24T14:32:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4665000000000003e-05 2024-04-24T14:32:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421512+0000", 2024-04-24T14:32:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9619000000000001e-05 2024-04-24T14:32:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424198+0000", 2024-04-24T14:32:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026854610000000001 2024-04-24T14:32:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424310+0000", 2024-04-24T14:32:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011208 2024-04-24T14:32:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:87 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'57, mlcod=13'51)", 2024-04-24T14:32:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:32:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.195826237999999, 2024-04-24T14:32:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002974057, 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425542+0000", 2024-04-24T14:32:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425614+0000", 2024-04-24T14:32:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1328999999999994e-05 2024-04-24T14:32:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425664+0000", 2024-04-24T14:32:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0491999999999997e-05 2024-04-24T14:32:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425676+0000", 2024-04-24T14:32:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.134e-05 2024-04-24T14:32:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425798+0000", 2024-04-24T14:32:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012221700000000001 2024-04-24T14:32:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425860+0000", 2024-04-24T14:32:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1761999999999996e-05 2024-04-24T14:32:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428368+0000", 2024-04-24T14:32:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025084 2024-04-24T14:32:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428518+0000", 2024-04-24T14:32:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149884 2024-04-24T14:32:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-04-24T14:32:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:32:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.995717512999999, 2024-04-24T14:32:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002919694, 2024-04-24T14:32:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:32:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:32:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625651+0000", 2024-04-24T14:32:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625667+0000", 2024-04-24T14:32:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5973000000000001e-05 2024-04-24T14:32:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625689+0000", 2024-04-24T14:32:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:32:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625705+0000", 2024-04-24T14:32:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5797000000000001e-05 2024-04-24T14:32:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625805+0000", 2024-04-24T14:32:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9238000000000001e-05 2024-04-24T14:32:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625837+0000", 2024-04-24T14:32:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2722000000000003e-05 2024-04-24T14:32:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628476+0000", 2024-04-24T14:32:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026391980000000002 2024-04-24T14:32:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628572+0000", 2024-04-24T14:32:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5754000000000005e-05 2024-04-24T14:32:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'88)", 2024-04-24T14:32:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:32:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.555779635, 2024-04-24T14:32:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031994179999999999, 2024-04-24T14:32:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065589+0000", 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065602+0000", 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2737e-05 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065620+0000", 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8546000000000002e-05 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065629+0000", 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1600000000000004e-06 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065729+0000", 2024-04-24T14:32:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9933999999999996e-05 2024-04-24T14:32:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065792+0000", 2024-04-24T14:32:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2712000000000005e-05 2024-04-24T14:32:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068667+0000", 2024-04-24T14:32:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028752920000000002 2024-04-24T14:32:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068790+0000", 2024-04-24T14:32:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012244400000000001 2024-04-24T14:32:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:137 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'91)", 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.554423899, 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052524399999999997, 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:32:45.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066945+0000", 2024-04-24T14:32:45.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067031+0000", 2024-04-24T14:32:45.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6228000000000002e-05 2024-04-24T14:32:45.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067060+0000", 2024-04-24T14:32:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8445999999999998e-05 2024-04-24T14:32:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067069+0000", 2024-04-24T14:32:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999996e-06 2024-04-24T14:32:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067172+0000", 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010285099999999999 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067234+0000", 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2068000000000003e-05 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072090+0000", 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048560859999999999 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072199+0000", 2024-04-24T14:32:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108384 2024-04-24T14:32:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:138 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'96, mlcod=13'91)", 2024-04-24T14:32:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:32:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.553424135, 2024-04-24T14:32:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053159380000000001, 2024-04-24T14:32:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:32:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:32:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067945+0000", 2024-04-24T14:32:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067957+0000", 2024-04-24T14:32:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1963e-05 2024-04-24T14:32:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067982+0000", 2024-04-24T14:32:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5570000000000001e-05 2024-04-24T14:32:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067994+0000", 2024-04-24T14:32:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2306e-05 2024-04-24T14:32:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068282+0000", 2024-04-24T14:32:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028799000000000001 2024-04-24T14:32:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068307+0000", 2024-04-24T14:32:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4351000000000001e-05 2024-04-24T14:32:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073142+0000", 2024-04-24T14:32:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004835326 2024-04-24T14:32:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073262+0000", 2024-04-24T14:32:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119695 2024-04-24T14:32:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:139 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.551941308, 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039447479999999997, 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:32:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069428+0000", 2024-04-24T14:32:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069440+0000", 2024-04-24T14:32:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2047e-05 2024-04-24T14:32:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069469+0000", 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8852000000000001e-05 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069477+0000", 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069531+0000", 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4126999999999999e-05 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069559+0000", 2024-04-24T14:32:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7881e-05 2024-04-24T14:32:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073270+0000", 2024-04-24T14:32:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037104519999999999 2024-04-24T14:32:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073373+0000", 2024-04-24T14:32:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010380899999999999 2024-04-24T14:32:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:140 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'92)", 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.551115432, 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031722510000000001, 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:32:45.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:32:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070253+0000", 2024-04-24T14:32:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070307+0000", 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3909000000000002e-05 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070330+0000", 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2599000000000002e-05 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070337+0000", 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3030000000000002e-06 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070377+0000", 2024-04-24T14:32:45.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9916000000000002e-05 2024-04-24T14:32:45.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070398+0000", 2024-04-24T14:32:45.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0579000000000001e-05 2024-04-24T14:32:45.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073377+0000", 2024-04-24T14:32:45.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002979013 2024-04-24T14:32:45.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073427+0000", 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0078999999999999e-05 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:143 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'101, mlcod=13'98)", 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.544930151999999, 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030247590000000001, 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.701 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.702 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:32:45.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:32:45.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076438+0000", 2024-04-24T14:32:45.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076533+0000", 2024-04-24T14:32:45.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4443000000000004e-05 2024-04-24T14:32:45.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076563+0000", 2024-04-24T14:32:45.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0386999999999999e-05 2024-04-24T14:32:45.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076604+0000", 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1012e-05 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076711+0000", 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010637300000000001 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076737+0000", 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6190000000000002e-05 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079368+0000", 2024-04-24T14:32:45.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026313529999999999 2024-04-24T14:32:45.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079465+0000", 2024-04-24T14:32:45.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6426000000000001e-05 2024-04-24T14:32:45.705 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.705 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.705 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.705 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:159 1.0 e14/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'113, mlcod=14'109)", 2024-04-24T14:32:45.706 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:32:45.706 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.911110046000001, 2024-04-24T14:32:45.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052053589999999997, 2024-04-24T14:32:45.706 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.706 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.706 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:32:45.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:32:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710259+0000", 2024-04-24T14:32:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710272+0000", 2024-04-24T14:32:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.382e-05 2024-04-24T14:32:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710303+0000", 2024-04-24T14:32:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0629000000000001e-05 2024-04-24T14:32:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710322+0000", 2024-04-24T14:32:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8859999999999999e-05 2024-04-24T14:32:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710479+0000", 2024-04-24T14:32:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157076 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710535+0000", 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6289000000000003e-05 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715391+0000", 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048557139999999997 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715465+0000", 2024-04-24T14:32:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4301000000000006e-05 2024-04-24T14:32:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:161 1.0 e14/12 1:30306672:devicehealth::main.db.0000000000000000:head v 14'115, mlcod=14'112)", 2024-04-24T14:32:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:32:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.903999244000001, 2024-04-24T14:32:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039960450000000002, 2024-04-24T14:32:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:32:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:32:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717370+0000", 2024-04-24T14:32:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717499+0000", 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012877400000000001 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717513+0000", 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4229999999999999e-05 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717521+0000", 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718040+0000", 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051920099999999995 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718094+0000", 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859000000000001e-05 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721278+0000", 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031838560000000001 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721367+0000", 2024-04-24T14:32:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9241999999999997e-05 2024-04-24T14:32:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:45.714 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:45.715 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:32:45.715 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:32:45.715 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:32:45.716 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:32:45.716 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:32:45.716 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:32:45.717 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:32:45.717 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:32:45.717 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:32:45.717 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:45.659+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:45.752 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:45.755+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:45.853 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:45.859+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:45.861 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:32:45.862 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:45.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:32:45.954 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:45.955+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:45.969 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:45.969 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:45.969 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:32:45.969 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:32:45.969 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:45.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:32:46.055 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:46.058+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:46.080 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:32:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:32:46.081 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-04-24T14:32:46.081 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:46.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:32:46.158 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:46.158+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:46.192 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:32:46.192 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:46.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:32:46.259 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:46.258+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:46.348 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:46.349 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:46.349 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:32:46.349 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:32:46.349 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:46.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:32:46.360 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:46.363+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:46.432 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T14:32:46.459 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:46.459 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:32:46.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:32:46.459 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-04-24T14:32:46.459 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:46.460 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:46.462+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:46.469 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:32:46.562 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:46.562+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:46.578 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:46.579 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:46.579 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:32:46.579 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:46.588 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:32:46.662 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:46.663+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:46.687 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:46.687 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:46.687 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:32:46.687 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:32:46.687 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:46.696 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:32:46.763 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:46.766+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:46.794 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:46.795 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:32:46.795 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:32:46.795 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:32:46.795 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.795 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.795 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:32:46.795 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.473019461, 2024-04-24T14:32:46.795 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011830553000000001, 2024-04-24T14:32:46.795 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.795 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.795 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.795 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.795 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 36 2024-04-24T14:32:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:32:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:32:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.797 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.797 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.797 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326000+0000", 2024-04-24T14:32:46.797 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:46.797 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.797 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.797 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.797 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326006+0000", 2024-04-24T14:32:46.797 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9340000000000003e-06 2024-04-24T14:32:46.797 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.797 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.797 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.797 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326044+0000", 2024-04-24T14:32:46.798 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8124999999999998e-05 2024-04-24T14:32:46.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.798 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.798 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.798 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326054+0000", 2024-04-24T14:32:46.798 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9650000000000003e-06 2024-04-24T14:32:46.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.798 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.798 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.798 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332746+0000", 2024-04-24T14:32:46.798 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006692539 2024-04-24T14:32:46.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.798 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.799 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.799 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332833+0000", 2024-04-24T14:32:46.799 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6482000000000003e-05 2024-04-24T14:32:46.799 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.799 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.799 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.799 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332947+0000", 2024-04-24T14:32:46.799 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114472 2024-04-24T14:32:46.799 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.799 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.799 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.799 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334320+0000", 2024-04-24T14:32:46.799 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001372489 2024-04-24T14:32:46.800 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.800 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.800 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.800 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337717+0000", 2024-04-24T14:32:46.800 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003396849 2024-04-24T14:32:46.800 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.800 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.800 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.800 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337789+0000", 2024-04-24T14:32:46.800 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2080000000000001e-05 2024-04-24T14:32:46.800 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.800 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.800 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.800 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337831+0000", 2024-04-24T14:32:46.801 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.244e-05 2024-04-24T14:32:46.801 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.801 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.801 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.801 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.801 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.801 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.801 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:32:46.801 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.472957599000001, 2024-04-24T14:32:46.801 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:32:46.801 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.801 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.801 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.801 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.802 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.802 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:32:46.802 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.802 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.802 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.802 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.802 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:32:46.802 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.802 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.802 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.802 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.802 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:32:46.802 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.803 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.803 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.803 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.803 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:32:46.803 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:32:46.803 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.803 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.803 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.803 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:32:46.803 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:32:46.803 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.803 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.803 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.803 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:32:46.804 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:32:46.804 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.804 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.804 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.804 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:32:46.804 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:32:46.804 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.804 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.804 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.804 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:32:46.804 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:32:46.804 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.804 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.804 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.805 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:32:46.805 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:32:46.805 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.805 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.805 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.805 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:32:46.805 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:32:46.805 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.805 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.805 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.805 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:32:46.805 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:32:46.805 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.805 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.806 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.806 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:32:46.806 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:32:46.806 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.806 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.806 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.806 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:32:46.806 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:32:46.806 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.806 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.806 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.806 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:32:46.807 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:32:46.807 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.807 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.807 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.807 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.807 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.807 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:32:46.807 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.379453126000001, 2024-04-24T14:32:46.807 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011218514000000001, 2024-04-24T14:32:46.807 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.807 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 88 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:32:46.808 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.809 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.809 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.809 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.809 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419565+0000", 2024-04-24T14:32:46.809 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:46.809 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.809 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.809 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.809 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419575+0000", 2024-04-24T14:32:46.809 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5340000000000005e-06 2024-04-24T14:32:46.809 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.809 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.810 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419616+0000", 2024-04-24T14:32:46.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1301999999999999e-05 2024-04-24T14:32:46.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.810 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419638+0000", 2024-04-24T14:32:46.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2177000000000001e-05 2024-04-24T14:32:46.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.810 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428578+0000", 2024-04-24T14:32:46.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089398280000000004 2024-04-24T14:32:46.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.811 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.811 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428665+0000", 2024-04-24T14:32:46.811 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6644000000000006e-05 2024-04-24T14:32:46.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.811 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.811 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428797+0000", 2024-04-24T14:32:46.811 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013231500000000001 2024-04-24T14:32:46.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.811 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.811 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.429866+0000", 2024-04-24T14:32:46.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001069221 2024-04-24T14:32:46.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.812 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430684+0000", 2024-04-24T14:32:46.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081788399999999995 2024-04-24T14:32:46.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.812 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430743+0000", 2024-04-24T14:32:46.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8875999999999997e-05 2024-04-24T14:32:46.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.813 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.813 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430786+0000", 2024-04-24T14:32:46.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2398999999999998e-05 2024-04-24T14:32:46.813 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.813 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.813 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.813 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.813 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:32:46.813 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.173442656999999, 2024-04-24T14:32:46.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:32:46.813 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.814 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.814 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.814 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.814 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.814 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:32:46.814 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.814 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.814 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.814 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.814 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:32:46.814 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.814 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.814 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.815 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.815 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:32:46.815 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.815 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.815 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.815 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:32:46.815 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:46.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.815 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.815 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.815 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:32:46.815 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:32:46.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.816 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.816 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.816 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:32:46.816 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:32:46.816 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.816 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.816 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.816 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:32:46.816 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:32:46.816 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.816 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.816 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.816 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:32:46.817 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:32:46.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.817 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.817 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:32:46.817 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:32:46.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.817 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.817 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:32:46.817 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:32:46.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.817 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.818 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:32:46.818 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:32:46.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.818 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.818 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.818 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:32:46.818 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:32:46.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.818 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.818 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.818 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:32:46.818 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:32:46.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.819 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.819 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.819 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:32:46.819 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:32:46.819 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.819 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.819 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.819 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.819 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.819 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.819 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:32:46.819 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.173216888999999, 2024-04-24T14:32:46.819 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:32:46.819 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.820 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:32:46.821 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.821 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.821 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.821 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.821 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:32:46.821 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:46.821 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.821 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.821 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.821 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:32:46.821 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:32:46.821 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.821 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.822 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.822 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:32:46.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:32:46.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.822 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.822 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:32:46.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:32:46.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.822 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.822 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:32:46.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:32:46.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:32:46.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:32:46.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:32:46.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:32:46.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:32:46.824 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:32:46.824 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:32:46.824 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:32:46.824 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:32:46.824 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:32:46.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.825 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.825 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:32:46.825 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:32:46.825 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.825 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.825 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.825 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.825 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:32:46.825 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.173039802000002, 2024-04-24T14:32:46.826 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:32:46.826 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.826 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.826 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.826 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.826 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.826 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:32:46.826 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.826 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.826 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.826 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.826 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:32:46.826 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.827 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:32:46.827 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.827 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:32:46.827 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:46.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.827 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:32:46.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:32:46.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.828 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.828 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:32:46.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:32:46.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.828 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.828 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:32:46.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:32:46.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.828 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:32:46.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:32:46.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:32:46.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:32:46.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:32:46.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:32:46.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.830 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.830 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:32:46.830 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:32:46.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.830 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.830 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:32:46.830 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:32:46.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.830 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.830 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:32:46.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:32:46.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.831 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.831 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:32:46.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:32:46.831 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.831 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.831 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.831 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.831 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:32:46.832 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.172935888000001, 2024-04-24T14:32:46.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:32:46.832 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.832 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.832 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.832 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.832 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.832 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:32:46.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.832 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:32:46.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.833 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.833 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:32:46.833 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.833 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.833 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:32:46.833 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:32:46.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.833 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.834 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:32:46.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:32:46.834 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.834 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.834 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:32:46.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:32:46.834 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.834 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.834 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:32:46.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:32:46.834 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.835 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:32:46.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:32:46.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.835 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:32:46.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:32:46.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.835 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:32:46.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:32:46.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:32:46.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:32:46.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:32:46.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:32:46.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:32:46.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:32:46.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:32:46.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:32:46.837 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.837 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.837 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.837 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.837 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:32:46.838 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.172639568000001, 2024-04-24T14:32:46.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:32:46.838 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.838 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.838 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.838 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.838 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.838 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:32:46.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.838 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:32:46.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.839 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.839 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:32:46.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.839 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.839 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:32:46.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:32:46.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:32:46.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:32:46.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:32:46.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:32:46.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:32:46.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:32:46.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:32:46.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:32:46.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:32:46.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:32:46.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:32:46.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:32:46.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:32:46.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:32:46.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:32:46.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:32:46.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:32:46.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:32:46.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:32:46.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:32:46.843 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.843 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.843 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.844 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.844 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:32:46.844 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.172547436999999, 2024-04-24T14:32:46.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:32:46.844 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.844 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.844 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.844 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.844 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.844 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:32:46.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.844 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:32:46.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:32:46.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:32:46.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:46.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:32:46.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:32:46.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:32:46.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:32:46.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:32:46.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:32:46.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:32:46.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:32:46.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:32:46.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:32:46.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:32:46.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:32:46.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:32:46.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:32:46.849 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.849 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.849 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.850 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.850 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:32:46.850 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.172462225, 2024-04-24T14:32:46.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:32:46.850 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.850 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.850 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.850 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.850 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.850 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:32:46.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.850 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:32:46.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:32:46.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:32:46.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:46.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:32:46.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:32:46.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:32:46.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:32:46.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:32:46.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:32:46.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:32:46.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:32:46.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:32:46.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:32:46.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:32:46.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:32:46.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:32:46.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:32:46.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:32:46.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:32:46.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:32:46.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:32:46.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:32:46.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:32:46.855 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.855 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.855 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.856 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.856 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:32:46.856 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.172225344000001, 2024-04-24T14:32:46.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:32:46.856 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.856 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.856 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.856 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.856 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.856 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:32:46.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.856 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:32:46.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:32:46.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:32:46.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:32:46.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:32:46.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:32:46.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:32:46.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:32:46.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:32:46.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:32:46.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:32:46.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:32:46.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:32:46.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:32:46.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:32:46.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:32:46.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:32:46.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:32:46.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:32:46.861 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.861 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.861 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.862 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.862 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:32:46.862 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 24.172143821999999, 2024-04-24T14:32:46.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:32:46.862 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.862 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.862 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.862 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.862 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.862 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:32:46.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.863 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:32:46.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:32:46.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:32:46.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:46.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:32:46.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:32:46.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:32:46.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:32:46.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:32:46.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:32:46.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:32:46.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:32:46.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:32:46.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:32:46.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:32:46.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:32:46.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:32:46.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:32:46.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:32:46.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:32:46.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:32:46.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:32:46.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:32:46.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:32:46.868 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.868 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.868 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.868 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.868 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:32:46.868 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.740424745999999, 2024-04-24T14:32:46.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01155111, 2024-04-24T14:32:46.868 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.868 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.869 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.869 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.869 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.869 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135 2024-04-24T14:32:46.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.869 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:32:46.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:32:46.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:32:46.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:46.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058602+0000", 2024-04-24T14:32:46.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1239999999999999e-06 2024-04-24T14:32:46.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058609+0000", 2024-04-24T14:32:46.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6719999999999998e-06 2024-04-24T14:32:46.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058626+0000", 2024-04-24T14:32:46.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7569999999999999e-05 2024-04-24T14:32:46.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065271+0000", 2024-04-24T14:32:46.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00664469 2024-04-24T14:32:46.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065398+0000", 2024-04-24T14:32:46.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012694299999999999 2024-04-24T14:32:46.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065598+0000", 2024-04-24T14:32:46.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000200216 2024-04-24T14:32:46.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067596+0000", 2024-04-24T14:32:46.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019976450000000001 2024-04-24T14:32:46.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070055+0000", 2024-04-24T14:32:46.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024593710000000001 2024-04-24T14:32:46.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070109+0000", 2024-04-24T14:32:46.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3690999999999998e-05 2024-04-24T14:32:46.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070147+0000", 2024-04-24T14:32:46.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7910000000000001e-05 2024-04-24T14:32:46.874 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.874 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.874 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.874 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.874 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:32:46.874 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.740385410999998, 2024-04-24T14:32:46.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:32:46.874 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:32:46.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:32:46.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:32:46.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:32:46.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:32:46.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:32:46.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:32:46.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:32:46.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:32:46.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:32:46.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:32:46.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:32:46.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:32:46.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:32:46.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:32:46.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:32:46.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:32:46.880 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.880 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.880 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.880 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.880 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:32:46.880 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.740343908, 2024-04-24T14:32:46.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:32:46.880 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.880 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.880 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.880 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:32:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:32:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:32:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:32:46.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:32:46.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:32:46.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:32:46.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:32:46.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:32:46.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:32:46.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:32:46.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:32:46.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:32:46.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:32:46.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:32:46.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:32:46.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:32:46.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:32:46.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:32:46.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:32:46.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:32:46.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:32:46.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:32:46.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:32:46.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:32:46.886 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.886 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.886 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.886 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.886 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:32:46.886 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.740231475000002, 2024-04-24T14:32:46.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:32:46.886 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.886 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.887 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.887 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.887 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.887 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:32:46.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.887 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:32:46.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:32:46.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:32:46.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:46.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:32:46.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:32:46.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:32:46.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:32:46.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:32:46.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:32:46.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:32:46.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:32:46.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:32:46.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:32:46.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:32:46.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:32:46.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:32:46.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:32:46.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:32:46.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:32:46.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:32:46.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:32:46.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:32:46.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:32:46.892 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.892 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.892 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.892 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.892 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:32:46.892 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.740203175000001, 2024-04-24T14:32:46.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:32:46.892 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.893 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.893 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.893 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.893 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.893 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:32:46.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.893 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:32:46.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:32:46.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:32:46.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:32:46.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:32:46.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:32:46.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:32:46.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:32:46.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:32:46.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:32:46.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:32:46.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:32:46.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:32:46.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:32:46.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:32:46.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:32:46.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:32:46.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:32:46.898 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.898 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.898 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.898 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.898 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:32:46.898 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.740115470999999, 2024-04-24T14:32:46.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:32:46.898 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.898 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.898 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.899 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.899 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.899 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:32:46.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.899 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:32:46.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:32:46.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:32:46.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:32:46.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:32:46.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:32:46.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:32:46.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:32:46.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:32:46.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:32:46.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:32:46.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:32:46.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:32:46.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:32:46.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:32:46.903 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.904 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.904 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.904 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:46.904 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:32:46.904 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.739992669999999, 2024-04-24T14:32:46.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:32:46.904 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.904 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.904 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.904 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.904 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.905 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:32:46.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.905 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:32:46.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:32:46.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:32:46.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:46.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:32:46.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:32:46.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:32:46.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:32:46.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:32:46.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:32:46.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:32:46.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:32:46.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:32:46.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:32:46.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:32:46.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:32:46.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:32:46.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:32:46.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:32:46.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:32:46.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:32:46.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:32:46.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:32:46.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:32:46.909 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.909 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.910 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.910 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:32:46.910 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:32:46.910 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.096752737999999, 2024-04-24T14:32:46.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:32:46.910 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:46.910 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:46.910 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:46.910 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:46.910 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:46.910 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:46.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:32:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:32:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:32:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:32:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:32:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:32:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:46.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:32:46.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:32:46.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:46.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:32:46.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:32:46.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:46.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:32:46.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:32:46.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:46.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:32:46.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:32:46.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:46.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:32:46.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:32:46.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:46.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.915 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:46.916 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:46.916 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:46.917 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:32:46.918 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:46.918+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:47.017 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:47.017 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:47.017 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:32:47.017 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:47.018 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:47.019+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:47.026 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:32:47.118 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:47.118+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:47.127 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:47.127 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:32:47.128 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:32:47.128 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:47.137 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:32:47.219 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:47.218+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:47.238 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:47.238 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:32:47.239 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:32:47.239 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-24T14:32:47.239 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:47.247 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:32:47.319 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:47.322+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:47.345 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:47.345 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:47.345 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:32:47.345 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:47.354 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:32:47.420 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:47.422+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:47.432 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-24T14:32:47.458 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:47.458 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:47.459 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:32:47.459 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:32:47.459 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:47.468 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:32:47.521 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:47.523+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:47.565 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:47.565 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:32:47.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:32:47.566 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-24T14:32:47.566 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:47.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:32:47.621 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:47.622+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:47.685 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:47.685 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:47.685 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:32:47.685 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:47.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:32:47.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:32:47.739 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:47.742+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:47.794 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:47.795 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:47.795 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:32:47.795 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:32:47.795 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:47.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:32:47.823 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:47.827+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:47.864 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:32:47.885 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:32:47.900 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:47.900 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:47.906 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:32:47.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:32:47.923 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:47.926+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:47.931 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:32:47.966 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:32:47.999 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:32:48.001 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:32:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:32:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:32:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:32:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:32:48.002 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:32:48.024 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:48.024 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:48.024 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:48.026+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:48.033 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:32:48.125 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:48.126+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:48.130 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:48.131 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:48.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:32:48.226 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:48.226+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:48.286 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:48.286 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:48.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:32:48.339 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:48.342+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:48.396 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:48.396 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:48.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:32:48.428 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:48.430+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:48.505 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:48.505 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:48.514 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:32:48.529 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:48.530+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:48.608 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:48.608 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:48.617 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:32:48.663 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:48.667+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:48.727 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:48.727 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:48.730 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:48.731+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:48.736 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:32:48.831 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:48.835+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:48.836 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:48.836 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:48.845 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:32:48.932 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:48.934+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:48.945 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:48.945 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:48.955 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:32:49.033 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:49.034+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:49.065 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:49.065 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:49.074 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:32:49.134 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:49.135+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:49.171 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:49.171 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:49.180 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:32:49.235 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:49.238+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:49.323 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:49.323 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:49.332 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:32:49.336 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:49.339+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:49.431 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:49.431 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:49.436 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:49.439+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:49.440 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:32:49.446 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T14:32:49.537 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:49.539+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:49.582 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:49.583 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:49.592 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:32:49.638 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:49.638+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:49.689 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:49.690 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:49.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:32:49.738 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:49.738+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:49.793 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T14:32:49.793 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:32:49.793 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-24T14:32:49.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T14:32:49.796 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:49.796 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:49.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:32:49.851 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:49.854+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:49.940 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:49.942+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:49.947 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:49.947 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:49.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:32:50.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:32:50.041 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:50.043+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:50.054 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:50.054 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:50.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:32:50.121 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:32:50.121 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:50.121 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 27, 2024-04-24T14:32:50.121 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "6706935e-d09a-472b-b839-321319fa9e54", 2024-04-24T14:32:50.121 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2024-04-24T14:32:09.059925+0000", 2024-04-24T14:32:50.121 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2024-04-24T14:32:47.421309+0000", 2024-04-24T14:32:50.121 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2024-04-24T14:32:20.236258+0000", 2024-04-24T14:32:50.121 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2024-04-24T14:32:11.347028+0000", 2024-04-24T14:32:50.121 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T14:32:50.121 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5806080, 2024-04-24T14:32:50.121 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2024-04-24T14:32:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: "nodeep-scrub", 2024-04-24T14:32:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: "noscrub", 2024-04-24T14:32:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2024-04-24T14:32:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2024-04-24T14:32:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2024-04-24T14:32:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2024-04-24T14:32:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:32:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 4, 2024-04-24T14:32:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T14:32:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T14:32:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T14:32:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2024-04-24T14:32:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 3, 2024-04-24T14:32:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 6, 2024-04-24T14:32:50.123 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2024-04-24T14:32:50.123 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "jewel", 2024-04-24T14:32:50.123 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "quincy", 2024-04-24T14:32:50.123 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2024-04-24T14:32:50.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.123 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2024-04-24T14:32:50.123 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2024-04-24T14:32:50.123 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:21.233022+0000", 2024-04-24T14:32:50.123 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-24T14:32:50.123 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-24T14:32:50.123 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:32:50.123 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:32:50.123 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:32:50.124 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:32:50.124 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:32:50.124 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:32:50.124 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:32:50.124 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:32:50.124 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:32:50.124 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:32:50.124 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2024-04-24T14:32:50.124 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2024-04-24T14:32:50.124 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2024-04-24T14:32:50.124 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2024-04-24T14:32:50.124 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2024-04-24T14:32:50.124 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:32:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:32:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:32:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:32:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:32:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:32:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:32:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "14", 2024-04-24T14:32:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-24T14:32:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T14:32:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T14:32:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:32:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:32:50.125 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-24T14:32:50.126 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-24T14:32:50.126 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:32:50.126 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:32:50.126 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:32:50.126 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:32:50.126 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:32:50.126 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:32:50.126 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:32:50.126 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:32:50.126 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:32:50.126 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:32:50.126 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:32:50.126 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:32:50.127 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:32:50.127 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:32:50.127 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:32:50.127 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:32:50.127 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:32:50.127 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:32:50.127 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:32:50.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.127 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:32:50.127 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:32:50.127 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:32:50.127 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:32:50.127 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:32:50.128 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:32:50.128 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:32:50.128 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:32:50.128 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:32:50.128 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:32:50.128 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:32:50.128 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-24T14:32:50.128 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2024-04-24T14:32:50.128 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2024-04-24T14:32:50.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.128 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:32:50.128 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2024-04-24T14:32:50.128 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.129 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-24T14:32:50.129 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2024-04-24T14:32:50.129 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:26.609964+0000", 2024-04-24T14:32:50.129 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-24T14:32:50.129 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-24T14:32:50.129 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:32:50.129 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:32:50.129 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:32:50.129 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:32:50.129 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:32:50.129 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:32:50.129 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:32:50.130 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:32:50.130 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:32:50.130 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:32:50.130 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2024-04-24T14:32:50.130 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2024-04-24T14:32:50.130 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2024-04-24T14:32:50.130 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2024-04-24T14:32:50.130 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2024-04-24T14:32:50.130 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:32:50.130 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:32:50.130 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:32:50.130 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:32:50.130 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:32:50.131 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:32:50.131 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:32:50.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.131 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "21", 2024-04-24T14:32:50.131 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "21", 2024-04-24T14:32:50.131 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "21", 2024-04-24T14:32:50.131 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "21", 2024-04-24T14:32:50.131 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:32:50.131 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:32:50.131 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-24T14:32:50.131 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 19, 2024-04-24T14:32:50.131 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:32:50.131 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:32:50.132 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:32:50.132 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:32:50.132 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [ 2024-04-24T14:32:50.132 INFO:teuthology.orchestra.run.smithi090.stdout: 3 2024-04-24T14:32:50.132 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:32:50.132 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:32:50.132 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": 3, 2024-04-24T14:32:50.132 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": 3, 2024-04-24T14:32:50.132 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:32:50.132 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:32:50.132 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:32:50.132 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:32:50.132 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:32:50.133 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:32:50.133 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:32:50.133 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:32:50.133 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:32:50.133 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:32:50.133 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:32:50.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.133 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:32:50.133 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:32:50.133 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:32:50.133 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-24T14:32:50.134 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "cache", 2024-04-24T14:32:50.135 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:33.987564+0000", 2024-04-24T14:32:50.135 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 9, 2024-04-24T14:32:50.135 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,incomplete_clones", 2024-04-24T14:32:50.135 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:32:50.135 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:32:50.135 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:32:50.135 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:32:50.135 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:32:50.135 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:32:50.135 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:32:50.135 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:32:50.135 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:32:50.135 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:32:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 4, 2024-04-24T14:32:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 4, 2024-04-24T14:32:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 4, 2024-04-24T14:32:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 4, 2024-04-24T14:32:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 4, 2024-04-24T14:32:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:32:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:32:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:32:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:32:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:32:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:32:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:32:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.137 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "25", 2024-04-24T14:32:50.137 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "21", 2024-04-24T14:32:50.137 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "21", 2024-04-24T14:32:50.137 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "21", 2024-04-24T14:32:50.137 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:32:50.137 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:32:50.137 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-24T14:32:50.137 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 19, 2024-04-24T14:32:50.137 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:32:50.137 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:32:50.137 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:32:50.137 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:32:50.137 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": 2, 2024-04-24T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "writeback", 2024-04-24T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 250, 2024-04-24T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bloom", 2024-04-24T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: "false_positive_probability": 0.050000000000000003, 2024-04-24T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: "target_size": 0, 2024-04-24T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: "seed": 0 2024-04-24T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 60, 2024-04-24T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 8, 2024-04-24T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [ 2024-04-24T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000 2024-04-24T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-24T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-24T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "a2506b4f-3486-4f51-972f-1d9e4b015346", 2024-04-24T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 18, 2024-04-24T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2024-04-24T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2024-04-24T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2024-04-24T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2024-04-24T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2024-04-24T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2024-04-24T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6817/20309", 2024-04-24T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6819/20309", 2024-04-24T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/20309", 2024-04-24T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6821/20309", 2024-04-24T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-24T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "9e17324f-a615-494f-ba0d-7c8aa8dac95c", 2024-04-24T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 18, 2024-04-24T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2024-04-24T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2024-04-24T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2024-04-24T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2024-04-24T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2024-04-24T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2024-04-24T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2024-04-24T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2024-04-24T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/20308", 2024-04-24T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/20308", 2024-04-24T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/20308", 2024-04-24T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/20308", 2024-04-24T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-24T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7", 2024-04-24T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 15, 2024-04-24T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2024-04-24T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2024-04-24T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2024-04-24T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2024-04-24T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2024-04-24T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2024-04-24T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2024-04-24T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2024-04-24T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6809/20310", 2024-04-24T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6811/20310", 2024-04-24T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6815/20310", 2024-04-24T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6813/20310", 2024-04-24T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-24T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "509c16da-22b2-48e8-aa51-c320c8c4fd8b", 2024-04-24T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 15, 2024-04-24T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6800", 2024-04-24T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6801", 2024-04-24T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6802", 2024-04-24T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6803", 2024-04-24T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:32:50.162 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.162 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.162 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6806", 2024-04-24T14:32:50.162 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:32:50.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.162 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.162 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6807", 2024-04-24T14:32:50.162 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:32:50.162 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.162 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.163 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:32:50.163 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.163 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.163 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6804", 2024-04-24T14:32:50.163 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:32:50.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.163 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.163 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6805", 2024-04-24T14:32:50.163 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:32:50.163 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.163 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.164 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6801/20316", 2024-04-24T14:32:50.164 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6803/20316", 2024-04-24T14:32:50.164 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6807/20316", 2024-04-24T14:32:50.164 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6805/20316", 2024-04-24T14:32:50.164 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:32:50.164 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:32:50.164 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:32:50.164 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.164 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-24T14:32:50.164 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1462066b-4e59-46d4-9b85-1f61ee1d56a6", 2024-04-24T14:32:50.164 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:32:50.164 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:32:50.165 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:32:50.165 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:32:50.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:32:50.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:32:50.165 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:32:50.165 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 18, 2024-04-24T14:32:50.165 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:32:50.165 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:32:50.165 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:32:50.165 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.165 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.165 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6808", 2024-04-24T14:32:50.166 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:32:50.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.166 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.166 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6809", 2024-04-24T14:32:50.166 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:32:50.166 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.166 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.166 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:32:50.166 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.166 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.166 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6810", 2024-04-24T14:32:50.167 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:32:50.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.167 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.167 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6811", 2024-04-24T14:32:50.167 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:32:50.167 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.167 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.167 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:32:50.167 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.167 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.167 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6814", 2024-04-24T14:32:50.168 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:32:50.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.168 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.168 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6815", 2024-04-24T14:32:50.168 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:32:50.168 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.168 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.168 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:32:50.168 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.168 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.168 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6812", 2024-04-24T14:32:50.169 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:32:50.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.169 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.169 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6813", 2024-04-24T14:32:50.169 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:32:50.169 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.169 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.169 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6809/20318", 2024-04-24T14:32:50.169 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6811/20318", 2024-04-24T14:32:50.169 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6815/20318", 2024-04-24T14:32:50.169 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6813/20318", 2024-04-24T14:32:50.170 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:32:50.170 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:32:50.170 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:32:50.170 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.170 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-24T14:32:50.170 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "4dc954a8-5f3c-4826-b9bc-98cf30519382", 2024-04-24T14:32:50.170 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:32:50.170 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:32:50.170 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:32:50.170 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:32:50.170 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:32:50.171 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:32:50.171 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:32:50.171 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 15, 2024-04-24T14:32:50.171 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:32:50.171 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:32:50.171 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:32:50.171 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.171 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.171 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6816", 2024-04-24T14:32:50.171 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:32:50.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6817", 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6818", 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.172 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.173 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6819", 2024-04-24T14:32:50.173 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:32:50.173 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.173 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.173 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:32:50.173 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.173 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.173 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6822", 2024-04-24T14:32:50.173 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:32:50.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.174 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.174 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6823", 2024-04-24T14:32:50.174 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:32:50.174 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.174 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.174 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:32:50.174 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:32:50.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.174 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:32:50.174 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6820", 2024-04-24T14:32:50.174 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:32:50.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.175 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:32:50.175 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6821", 2024-04-24T14:32:50.175 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:32:50.175 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.175 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.175 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6817/20317", 2024-04-24T14:32:50.175 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6819/20317", 2024-04-24T14:32:50.175 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6823/20317", 2024-04-24T14:32:50.175 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6821/20317", 2024-04-24T14:32:50.175 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:32:50.175 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:32:50.175 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:32:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:32:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2024-04-24T14:32:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-24T14:32:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:32:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:32:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:32:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:32:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:32:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.792899+0000", 2024-04-24T14:32:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:32:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-24T14:32:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:32:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:32:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:32:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:32:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:32:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.580414+0000", 2024-04-24T14:32:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:32:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-24T14:32:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:32:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:32:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:32:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:32:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:32:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.713336+0000", 2024-04-24T14:32:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:32:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-24T14:32:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:32:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:32:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:32:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:32:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:32:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.579638+0000", 2024-04-24T14:32:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:32:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-24T14:32:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:32:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:32:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:32:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:32:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:32:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.570510+0000", 2024-04-24T14:32:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:32:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-24T14:32:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:32:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:32:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:32:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:32:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:32:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.736558+0000", 2024-04-24T14:32:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:32:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:32:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2024-04-24T14:32:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [], 2024-04-24T14:32:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2024-04-24T14:32:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2024-04-24T14:32:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [], 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2024-04-24T14:32:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2024-04-24T14:32:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2024-04-24T14:32:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2024-04-24T14:32:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2024-04-24T14:32:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2024-04-24T14:32:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2024-04-24T14:32:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2024-04-24T14:32:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:50.182 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:50.184 INFO:tasks.thrashosds.thrasher:j is {'epoch': 27, 'fsid': '6706935e-d09a-472b-b839-321319fa9e54', 'created': '2024-04-24T14:32:09.059925+0000', 'modified': '2024-04-24T14:32:47.421309+0000', 'last_up_change': '2024-04-24T14:32:20.236258+0000', 'last_in_change': '2024-04-24T14:32:11.347028+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T14:32:21.233022+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T14:32:26.609964+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '21', 'last_force_op_resend_prenautilus': '21', 'last_force_op_resend_preluminous': '21', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-04-24T14:32:33.987564+0000', 'flags': 9, 'flags_names': 'hashpspool,incomplete_clones', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '21', 'last_force_op_resend_prenautilus': '21', 'last_force_op_resend_preluminous': '21', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 2, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'a2506b4f-3486-4f51-972f-1d9e4b015346', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 20309}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6819', 'nonce': 20309}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 20309}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6820', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 20309}]}, 'public_addr': '172.21.15.90:6817/20309', 'cluster_addr': '172.21.15.90:6819/20309', 'heartbeat_back_addr': '172.21.15.90:6823/20309', 'heartbeat_front_addr': '172.21.15.90:6821/20309', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9e17324f-a615-494f-ba0d-7c8aa8dac95c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 20308}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 20308}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 20308}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 20308}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 20308}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 20308}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 20308}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 20308}]}, 'public_addr': '172.21.15.90:6801/20308', 'cluster_addr': '172.21.15.90:6803/20308', 'heartbeat_back_addr': '172.21.15.90:6807/20308', 'heartbeat_front_addr': '172.21.15.90:6805/20308', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 20310}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6810', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 20310}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6814', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 20310}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 20310}]}, 'public_addr': '172.21.15.90:6809/20310', 'cluster_addr': '172.21.15.90:6811/20310', 'heartbeat_back_addr': '172.21.15.90:6815/20310', 'heartbeat_front_addr': '172.21.15.90:6813/20310', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '509c16da-22b2-48e8-aa51-c320c8c4fd8b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6800', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6801', 'nonce': 20316}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6802', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6803', 'nonce': 20316}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6806', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6807', 'nonce': 20316}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6804', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6805', 'nonce': 20316}]}, 'public_addr': '172.21.15.103:6801/20316', 'cluster_addr': '172.21.15.103:6803/20316', 'heartbeat_back_addr': '172.21.15.103:6807/20316', 'heartbeat_front_addr': '172.21.15.103:6805/20316', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1462066b-4e59-46d4-9b85-1f61ee1d56a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6808', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6809', 'nonce': 20318}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6810', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6811', 'nonce': 20318}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6814', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6815', 'nonce': 20318}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6812', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6813', 'nonce': 20318}]}, 'public_addr': '172.21.15.103:6809/20318', 'cluster_addr': '172.21.15.103:6811/20318', 'heartbeat_back_addr': '172.21.15.103:6815/20318', 'heartbeat_front_addr': '172.21.15.103:6813/20318', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4dc954a8-5f3c-4826-b9bc-98cf30519382', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6816', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6817', 'nonce': 20317}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6818', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6819', 'nonce': 20317}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6822', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6823', 'nonce': 20317}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6820', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6821', 'nonce': 20317}]}, 'public_addr': '172.21.15.103:6817/20317', 'cluster_addr': '172.21.15.103:6819/20317', 'heartbeat_back_addr': '172.21.15.103:6823/20317', 'heartbeat_front_addr': '172.21.15.103:6821/20317', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.792899+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.580414+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.713336+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.579638+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.570510+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.736558+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T14:32:50.184 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-24T14:32:50.184 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:32:50.185 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:32:50.185 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:50.187+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:50.198 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:32:50.206 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:32:50.206 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:50.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:32:50.219 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:32:50.242 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:32:50.262 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:32:50.264 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:32:50.264 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:32:50.265 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:32:50.265 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:32:50.265 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:32:50.265 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:32:50.311 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:32:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:32:50.311 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:50.312 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:50.315+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:50.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:32:50.385 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:50.387+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:50.446 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-24T14:32:50.471 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:32:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:32:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-04-24T14:32:50.471 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:50.480 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:32:50.485 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:50.487+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:50.580 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:32:50.581 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:50.586 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:50.591+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:50.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:32:50.687 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:50.691+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:50.690 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:50.690 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:50.690 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:32:50.690 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:32:50.690 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:50.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:32:50.788 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:50.791+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:50.800 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:50.800 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:32:50.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:32:50.800 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-04-24T14:32:50.801 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:50.809 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:32:50.888 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:50.891+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:50.920 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:32:50.920 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:50.929 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:32:50.989 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:50.991+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:51.024 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:51.025 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:51.025 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:32:51.025 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:32:51.025 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:51.034 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:32:51.090 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:51.091+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:51.131 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:51.131 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:32:51.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:32:51.131 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:32:51.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.131 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.131 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:32:51.131 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.809780195999998, 2024-04-24T14:32:51.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011830553000000001, 2024-04-24T14:32:51.131 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.132 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.132 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.132 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.132 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.132 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 36 2024-04-24T14:32:51.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.132 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:32:51.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:32:51.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326000+0000", 2024-04-24T14:32:51.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:51.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326006+0000", 2024-04-24T14:32:51.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9340000000000003e-06 2024-04-24T14:32:51.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326044+0000", 2024-04-24T14:32:51.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8124999999999998e-05 2024-04-24T14:32:51.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326054+0000", 2024-04-24T14:32:51.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9650000000000003e-06 2024-04-24T14:32:51.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332746+0000", 2024-04-24T14:32:51.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006692539 2024-04-24T14:32:51.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332833+0000", 2024-04-24T14:32:51.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6482000000000003e-05 2024-04-24T14:32:51.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332947+0000", 2024-04-24T14:32:51.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114472 2024-04-24T14:32:51.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334320+0000", 2024-04-24T14:32:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001372489 2024-04-24T14:32:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337717+0000", 2024-04-24T14:32:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003396849 2024-04-24T14:32:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337789+0000", 2024-04-24T14:32:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2080000000000001e-05 2024-04-24T14:32:51.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337831+0000", 2024-04-24T14:32:51.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.244e-05 2024-04-24T14:32:51.137 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.137 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.137 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.137 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.137 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:32:51.137 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.809718333999999, 2024-04-24T14:32:51.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:32:51.137 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.138 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.138 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.138 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.138 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.138 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:32:51.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.138 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:32:51.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:32:51.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:32:51.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:32:51.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:32:51.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:32:51.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:32:51.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:32:51.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:32:51.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:32:51.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:32:51.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:32:51.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:32:51.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:32:51.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:32:51.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:32:51.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:32:51.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:32:51.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:32:51.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:32:51.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:32:51.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:32:51.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:32:51.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:32:51.143 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.143 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.143 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.143 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.143 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:32:51.143 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.716213861, 2024-04-24T14:32:51.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011218514000000001, 2024-04-24T14:32:51.144 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.144 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.144 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.144 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.144 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.144 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 88 2024-04-24T14:32:51.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.144 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:32:51.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:32:51.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419565+0000", 2024-04-24T14:32:51.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:51.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419575+0000", 2024-04-24T14:32:51.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5340000000000005e-06 2024-04-24T14:32:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419616+0000", 2024-04-24T14:32:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1301999999999999e-05 2024-04-24T14:32:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419638+0000", 2024-04-24T14:32:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2177000000000001e-05 2024-04-24T14:32:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428578+0000", 2024-04-24T14:32:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089398280000000004 2024-04-24T14:32:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428665+0000", 2024-04-24T14:32:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6644000000000006e-05 2024-04-24T14:32:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428797+0000", 2024-04-24T14:32:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013231500000000001 2024-04-24T14:32:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.429866+0000", 2024-04-24T14:32:51.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001069221 2024-04-24T14:32:51.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430684+0000", 2024-04-24T14:32:51.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081788399999999995 2024-04-24T14:32:51.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430743+0000", 2024-04-24T14:32:51.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8875999999999997e-05 2024-04-24T14:32:51.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430786+0000", 2024-04-24T14:32:51.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2398999999999998e-05 2024-04-24T14:32:51.149 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.149 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.149 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.149 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.149 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:32:51.149 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.510203392000001, 2024-04-24T14:32:51.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:32:51.149 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.150 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.150 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.150 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.150 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.150 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:32:51.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.150 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:32:51.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:32:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:32:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:32:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:32:51.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:32:51.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:32:51.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:32:51.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:32:51.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:32:51.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:32:51.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:32:51.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:32:51.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:32:51.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:32:51.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:32:51.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:32:51.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:32:51.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:32:51.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:32:51.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:32:51.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:32:51.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:32:51.155 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.155 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.155 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.155 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.155 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:32:51.155 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.509977624000001, 2024-04-24T14:32:51.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:32:51.155 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.155 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.156 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.156 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.156 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.156 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:32:51.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.156 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:32:51.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:32:51.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:32:51.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:51.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:32:51.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:32:51.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:32:51.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:32:51.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:32:51.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:32:51.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:32:51.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:32:51.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:32:51.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:32:51.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:32:51.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:32:51.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:32:51.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:32:51.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:32:51.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:32:51.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:32:51.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:32:51.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:32:51.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:32:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:32:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.509800537, 2024-04-24T14:32:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:32:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.161 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.162 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:32:51.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.162 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:32:51.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:32:51.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:32:51.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:51.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:32:51.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:32:51.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:32:51.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:32:51.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:32:51.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:32:51.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:32:51.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:32:51.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.167 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.167 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:32:51.167 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.509696623, 2024-04-24T14:32:51.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:32:51.167 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.167 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.167 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.167 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.167 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.167 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:32:51.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.167 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:32:51.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:32:51.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:32:51.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:32:51.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:32:51.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:32:51.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:32:51.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:32:51.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:32:51.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:32:51.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:32:51.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:32:51.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:32:51.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:32:51.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:32:51.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:32:51.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:32:51.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:32:51.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:32:51.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:32:51.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:32:51.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:32:51.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:32:51.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:32:51.172 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.172 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.172 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.172 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.173 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:32:51.173 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.509400303, 2024-04-24T14:32:51.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:32:51.173 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.173 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.173 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.173 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.173 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.173 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:32:51.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.173 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:32:51.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:32:51.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:32:51.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:32:51.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:32:51.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:32:51.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:32:51.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:32:51.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:32:51.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:32:51.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:32:51.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:32:51.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:32:51.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:32:51.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:32:51.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:32:51.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:32:51.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:32:51.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:32:51.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:32:51.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:32:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:32:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:32:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:32:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.179 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.179 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:32:51.179 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.509308172000001, 2024-04-24T14:32:51.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:32:51.179 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.179 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.179 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.179 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.179 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.179 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:32:51.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.179 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:32:51.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:32:51.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:32:51.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:51.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:32:51.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:32:51.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:32:51.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:32:51.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:32:51.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:32:51.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:32:51.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:32:51.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:32:51.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:32:51.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:32:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:32:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:32:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:32:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.184 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:32:51.185 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.509222959999999, 2024-04-24T14:32:51.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:32:51.185 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.185 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.185 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.185 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.185 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.185 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:32:51.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.185 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:32:51.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:32:51.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:32:51.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:51.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:32:51.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:32:51.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:32:51.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:32:51.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:32:51.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:32:51.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:32:51.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.189 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:32:51.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:32:51.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.189 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:32:51.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:32:51.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:32:51.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:32:51.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.190 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:32:51.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:32:51.190 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.190 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.190 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.190 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.508986079, 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:32:51.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:32:51.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:32:51.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:32:51.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:32:51.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:32:51.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:32:51.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:32:51.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:32:51.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:32:51.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:32:51.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:32:51.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:32:51.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:32:51.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:32:51.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:32:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:32:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:32:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.508904557000001, 2024-04-24T14:32:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:32:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.196 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:32:51.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:32:51.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:51.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:32:51.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:32:51.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:32:51.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:32:51.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.200 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:32:51.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:32:51.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.200 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:32:51.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:32:51.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.201 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.201 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:32:51.201 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:32:51.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.201 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.201 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:32:51.201 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:32:51.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.201 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.201 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:32:51.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:32:51.202 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.202 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.202 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.202 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.202 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:32:51.202 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.077185481000001, 2024-04-24T14:32:51.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01155111, 2024-04-24T14:32:51.202 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.202 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.202 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:32:51.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:51.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058602+0000", 2024-04-24T14:32:51.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1239999999999999e-06 2024-04-24T14:32:51.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058609+0000", 2024-04-24T14:32:51.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6719999999999998e-06 2024-04-24T14:32:51.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.205 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058626+0000", 2024-04-24T14:32:51.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7569999999999999e-05 2024-04-24T14:32:51.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.205 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065271+0000", 2024-04-24T14:32:51.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00664469 2024-04-24T14:32:51.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.205 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065398+0000", 2024-04-24T14:32:51.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012694299999999999 2024-04-24T14:32:51.206 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.206 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.206 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.206 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065598+0000", 2024-04-24T14:32:51.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000200216 2024-04-24T14:32:51.206 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.206 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.206 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.206 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067596+0000", 2024-04-24T14:32:51.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019976450000000001 2024-04-24T14:32:51.206 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.206 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070055+0000", 2024-04-24T14:32:51.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024593710000000001 2024-04-24T14:32:51.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070109+0000", 2024-04-24T14:32:51.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3690999999999998e-05 2024-04-24T14:32:51.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070147+0000", 2024-04-24T14:32:51.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7910000000000001e-05 2024-04-24T14:32:51.208 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.208 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.208 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.208 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.208 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:32:51.208 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.077146146, 2024-04-24T14:32:51.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:32:51.208 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.208 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.208 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.208 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.208 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:32:51.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:32:51.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.210 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:32:51.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:32:51.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.210 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:32:51.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:32:51.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:32:51.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:32:51.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:32:51.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:32:51.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:32:51.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:32:51.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:32:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:32:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:32:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:32:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:32:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.214 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.214 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:32:51.214 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.077104642999998, 2024-04-24T14:32:51.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:32:51.214 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.214 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.214 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.214 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.214 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.214 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:32:51.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.215 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:32:51.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:32:51.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:32:51.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:32:51.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:32:51.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:32:51.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:32:51.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:32:51.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:32:51.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:32:51.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:32:51.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:32:51.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:32:51.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:32:51.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:32:51.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:32:51.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:32:51.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:32:51.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:32:51.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:32:51.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:32:51.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:32:51.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:32:51.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:32:51.219 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.219 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.219 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.220 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.220 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:32:51.220 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.07699221, 2024-04-24T14:32:51.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:32:51.220 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.220 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.220 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.220 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.220 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.220 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:32:51.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:32:51.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:51.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:32:51.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:32:51.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:32:51.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:32:51.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:32:51.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:32:51.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:32:51.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:32:51.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:32:51.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:32:51.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:32:51.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:32:51.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:32:51.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:32:51.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:32:51.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:32:51.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:32:51.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:32:51.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:32:51.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:32:51.225 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.225 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.225 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.225 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.226 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:32:51.226 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.07696391, 2024-04-24T14:32:51.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:32:51.226 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.226 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.226 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.226 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.226 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.226 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:32:51.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.226 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:32:51.227 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.227 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.227 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.227 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.227 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:32:51.227 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.227 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.227 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.227 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.227 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:32:51.227 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:32:51.227 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.227 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.227 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.228 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:32:51.228 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:32:51.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.228 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.228 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:32:51.228 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:32:51.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.228 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.228 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:32:51.228 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:32:51.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:32:51.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:32:51.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:32:51.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:32:51.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:32:51.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:32:51.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.230 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:32:51.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:32:51.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.230 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:32:51.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:32:51.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.231 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.232 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:32:51.232 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.076876206000001, 2024-04-24T14:32:51.232 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:32:51.232 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.232 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.232 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.232 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.232 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.232 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:32:51.232 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.232 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.232 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.232 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.233 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:32:51.233 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.233 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.233 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.233 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:32:51.233 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.233 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.233 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.233 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:32:51.233 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:51.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:32:51.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:32:51.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:32:51.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:32:51.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:32:51.235 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:32:51.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.235 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:32:51.235 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:32:51.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.235 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:32:51.235 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:32:51.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.236 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:32:51.236 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:32:51.236 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.236 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.236 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.236 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:32:51.236 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:32:51.236 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.236 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.236 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.236 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:32:51.236 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:32:51.236 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.236 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.237 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.237 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:32:51.237 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:32:51.237 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.237 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.237 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.237 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:32:51.237 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:32:51.237 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.237 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.237 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.237 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.237 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.237 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:51.238 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:32:51.238 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.076753405000002, 2024-04-24T14:32:51.238 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:32:51.238 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.238 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.238 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.238 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.238 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.238 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:32:51.238 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.238 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.238 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.238 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.238 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:32:51.239 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.239 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.239 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.239 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:32:51.239 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.239 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.239 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.239 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:32:51.239 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:51.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.239 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.239 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.240 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:32:51.240 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:32:51.240 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.240 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.240 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:32:51.240 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:32:51.240 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.240 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.240 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:32:51.240 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:32:51.240 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.241 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.241 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.241 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:32:51.241 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:32:51.241 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.241 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.241 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.241 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:32:51.241 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:32:51.241 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.241 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.241 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.241 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:32:51.241 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:32:51.242 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.242 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.242 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.242 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:32:51.242 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:32:51.242 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.242 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.242 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.242 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:32:51.242 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:32:51.242 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.242 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:32:51.243 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:32:51.244 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.433513473000001, 2024-04-24T14:32:51.244 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:32:51.244 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:51.244 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:51.244 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:51.244 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:51.244 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:51.244 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:32:51.244 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.244 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:51.244 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.244 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:51.244 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:32:51.244 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.245 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.245 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:51.245 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:32:51.245 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:51.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.245 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.245 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:51.245 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:32:51.245 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:32:51.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.245 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.245 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:51.246 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:32:51.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:32:51.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.246 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:51.246 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:32:51.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:32:51.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.246 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:51.246 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:32:51.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:32:51.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:51.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:32:51.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:32:51.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:51.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:32:51.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:32:51.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:51.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:32:51.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:32:51.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.248 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:51.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:32:51.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:32:51.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.248 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:51.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:32:51.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:32:51.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.248 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:51.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:32:51.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:32:51.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:51.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:51.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:51.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:32:51.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:32:51.249 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.249 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.249 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.249 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:51.249 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:51.249 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:51.251 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:32:51.251 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:51.255+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:51.351 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:51.355+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:51.401 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:51.401 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:51.401 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:32:51.402 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:51.410 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:32:51.455 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:51.459+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:51.511 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:51.511 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:51.511 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:32:51.511 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:32:51.511 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:51.519 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:32:51.553 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:51.555+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:51.612 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:51.612 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:32:51.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:32:51.612 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-24T14:32:51.612 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:51.620 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:32:51.653 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:51.655+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:51.714 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:51.714 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:51.714 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:32:51.714 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:51.723 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:32:51.754 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:51.755+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:51.816 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:51.816 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:51.816 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:32:51.816 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:32:51.817 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:51.825 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:32:51.855 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:51.859+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:51.920 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:32:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:32:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-24T14:32:51.921 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:51.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:32:51.975 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:51.979+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:52.041 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:52.041 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:32:52.041 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:52.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:32:52.057 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:52.059+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:52.157 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:32:52.157 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:32:52.157 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:52.158 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:52.159+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:52.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:32:52.259 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:52.263+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:52.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:32:52.313 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:32:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:32:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:32:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:36 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'22, mlcod=13'17)", 2024-04-24T14:32:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:32:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.984357889000002, 2024-04-24T14:32:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043791400000000001, 2024-04-24T14:32:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:32:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:32:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331851+0000", 2024-04-24T14:32:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331864+0000", 2024-04-24T14:32:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2767e-05 2024-04-24T14:32:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331881+0000", 2024-04-24T14:32:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6586e-05 2024-04-24T14:32:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331894+0000", 2024-04-24T14:32:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2976000000000001e-05 2024-04-24T14:32:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332161+0000", 2024-04-24T14:32:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026765399999999998 2024-04-24T14:32:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332197+0000", 2024-04-24T14:32:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5565999999999998e-05 2024-04-24T14:32:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336158+0000", 2024-04-24T14:32:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039616820000000002 2024-04-24T14:32:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336232+0000", 2024-04-24T14:32:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3151999999999994e-05 2024-04-24T14:32:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:37 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'23, mlcod=13'18)", 2024-04-24T14:32:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:32:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.983471411, 2024-04-24T14:32:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040417389999999999, 2024-04-24T14:32:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:32:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:32:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332738+0000", 2024-04-24T14:32:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332750+0000", 2024-04-24T14:32:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2432000000000001e-05 2024-04-24T14:32:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332768+0000", 2024-04-24T14:32:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7314000000000001e-05 2024-04-24T14:32:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332780+0000", 2024-04-24T14:32:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2716e-05 2024-04-24T14:32:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332925+0000", 2024-04-24T14:32:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144587 2024-04-24T14:32:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332955+0000", 2024-04-24T14:32:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0246000000000001e-05 2024-04-24T14:32:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336698+0000", 2024-04-24T14:32:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037430559999999998 2024-04-24T14:32:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336781+0000", 2024-04-24T14:32:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2447999999999994e-05 2024-04-24T14:32:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:39 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'25, mlcod=13'22)", 2024-04-24T14:32:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:32:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.977353088000001, 2024-04-24T14:32:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038736740000000001, 2024-04-24T14:32:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:32:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:32:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338856+0000", 2024-04-24T14:32:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338995+0000", 2024-04-24T14:32:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013868100000000001 2024-04-24T14:32:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339029+0000", 2024-04-24T14:32:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4372000000000002e-05 2024-04-24T14:32:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339050+0000", 2024-04-24T14:32:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0656999999999999e-05 2024-04-24T14:32:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339156+0000", 2024-04-24T14:32:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106177 2024-04-24T14:32:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339188+0000", 2024-04-24T14:32:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.243e-05 2024-04-24T14:32:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342648+0000", 2024-04-24T14:32:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034601689999999999 2024-04-24T14:32:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342731+0000", 2024-04-24T14:32:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2534999999999995e-05 2024-04-24T14:32:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:67 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'41, mlcod=13'39)", 2024-04-24T14:32:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:32:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.926616803999998, 2024-04-24T14:32:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031029009999999999, 2024-04-24T14:32:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:32:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:32:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389592+0000", 2024-04-24T14:32:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389603+0000", 2024-04-24T14:32:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090000000000001e-05 2024-04-24T14:32:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389631+0000", 2024-04-24T14:32:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7772000000000001e-05 2024-04-24T14:32:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389646+0000", 2024-04-24T14:32:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.432e-05 2024-04-24T14:32:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389732+0000", 2024-04-24T14:32:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6331999999999999e-05 2024-04-24T14:32:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389771+0000", 2024-04-24T14:32:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8751999999999998e-05 2024-04-24T14:32:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392585+0000", 2024-04-24T14:32:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028147469999999998 2024-04-24T14:32:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392696+0000", 2024-04-24T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111121 2024-04-24T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.925474192999999, 2024-04-24T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030475519999999998, 2024-04-24T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390735+0000", 2024-04-24T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390804+0000", 2024-04-24T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9488999999999998e-05 2024-04-24T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390832+0000", 2024-04-24T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7946e-05 2024-04-24T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390847+0000", 2024-04-24T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4693e-05 2024-04-24T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391508+0000", 2024-04-24T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066154799999999998 2024-04-24T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391538+0000", 2024-04-24T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0029e-05 2024-04-24T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393698+0000", 2024-04-24T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002160052 2024-04-24T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393784+0000", 2024-04-24T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5414999999999995e-05 2024-04-24T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:72 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'46, mlcod=13'43)", 2024-04-24T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.919300700000001, 2024-04-24T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036059460000000001, 2024-04-24T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396908+0000", 2024-04-24T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396968+0000", 2024-04-24T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9412e-05 2024-04-24T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397010+0000", 2024-04-24T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2169000000000002e-05 2024-04-24T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397022+0000", 2024-04-24T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1749999999999999e-05 2024-04-24T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397096+0000", 2024-04-24T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4701000000000002e-05 2024-04-24T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397126+0000", 2024-04-24T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2024-04-24T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400443+0000", 2024-04-24T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033172140000000002 2024-04-24T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400516+0000", 2024-04-24T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2181999999999994e-05 2024-04-24T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:73 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'47, mlcod=13'43)", 2024-04-24T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.918453295999999, 2024-04-24T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055152170000000002, 2024-04-24T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397756+0000", 2024-04-24T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397884+0000", 2024-04-24T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012842100000000001 2024-04-24T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397923+0000", 2024-04-24T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8942000000000001e-05 2024-04-24T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397939+0000", 2024-04-24T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6073e-05 2024-04-24T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398106+0000", 2024-04-24T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016737100000000001 2024-04-24T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398137+0000", 2024-04-24T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0496000000000001e-05 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403191+0000", 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00505391 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403272+0000", 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1483999999999994e-05 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:74 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'48, mlcod=13'43)", 2024-04-24T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.91765427, 2024-04-24T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048091760000000001, 2024-04-24T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398555+0000", 2024-04-24T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398570+0000", 2024-04-24T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.558e-05 2024-04-24T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398610+0000", 2024-04-24T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0028999999999999e-05 2024-04-24T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398625+0000", 2024-04-24T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-04-24T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398701+0000", 2024-04-24T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6507000000000005e-05 2024-04-24T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398731+0000", 2024-04-24T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9356000000000002e-05 2024-04-24T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403281+0000", 2024-04-24T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004549796 2024-04-24T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403365+0000", 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4685000000000007e-05 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.896341345, 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038140460000000002, 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419868+0000", 2024-04-24T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419882+0000", 2024-04-24T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.45e-05 2024-04-24T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419919+0000", 2024-04-24T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6939e-05 2024-04-24T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419961+0000", 2024-04-24T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2148999999999999e-05 2024-04-24T14:32:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420084+0000", 2024-04-24T14:32:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122958 2024-04-24T14:32:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420122+0000", 2024-04-24T14:32:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8334999999999998e-05 2024-04-24T14:32:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423616+0000", 2024-04-24T14:32:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003493566 2024-04-24T14:32:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423683+0000", 2024-04-24T14:32:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7149000000000004e-05 2024-04-24T14:32:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T14:32:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:32:52.353 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.89492959, 2024-04-24T14:32:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030290730000000002, 2024-04-24T14:32:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:32:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:32:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421279+0000", 2024-04-24T14:32:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421352+0000", 2024-04-24T14:32:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2251999999999999e-05 2024-04-24T14:32:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421378+0000", 2024-04-24T14:32:52.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6006000000000001e-05 2024-04-24T14:32:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421388+0000", 2024-04-24T14:32:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0373e-05 2024-04-24T14:32:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421473+0000", 2024-04-24T14:32:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4665000000000003e-05 2024-04-24T14:32:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421512+0000", 2024-04-24T14:32:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9619000000000001e-05 2024-04-24T14:32:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424198+0000", 2024-04-24T14:32:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026854610000000001 2024-04-24T14:32:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424310+0000", 2024-04-24T14:32:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011208 2024-04-24T14:32:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.357 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:87 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'57, mlcod=13'51)", 2024-04-24T14:32:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:32:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.890666630999998, 2024-04-24T14:32:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002974057, 2024-04-24T14:32:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:32:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:32:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425542+0000", 2024-04-24T14:32:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425614+0000", 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1328999999999994e-05 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425664+0000", 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0491999999999997e-05 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425676+0000", 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.134e-05 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425798+0000", 2024-04-24T14:32:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012221700000000001 2024-04-24T14:32:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425860+0000", 2024-04-24T14:32:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1761999999999996e-05 2024-04-24T14:32:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428368+0000", 2024-04-24T14:32:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025084 2024-04-24T14:32:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428518+0000", 2024-04-24T14:32:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149884 2024-04-24T14:32:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-04-24T14:32:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:32:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.690557905999999, 2024-04-24T14:32:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002919694, 2024-04-24T14:32:52.362 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625651+0000", 2024-04-24T14:32:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625667+0000", 2024-04-24T14:32:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5973000000000001e-05 2024-04-24T14:32:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625689+0000", 2024-04-24T14:32:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:32:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625705+0000", 2024-04-24T14:32:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5797000000000001e-05 2024-04-24T14:32:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625805+0000", 2024-04-24T14:32:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9238000000000001e-05 2024-04-24T14:32:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625837+0000", 2024-04-24T14:32:52.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2722000000000003e-05 2024-04-24T14:32:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628476+0000", 2024-04-24T14:32:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026391980000000002 2024-04-24T14:32:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628572+0000", 2024-04-24T14:32:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5754000000000005e-05 2024-04-24T14:32:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'88)", 2024-04-24T14:32:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:32:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.250620028, 2024-04-24T14:32:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031994179999999999, 2024-04-24T14:32:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:32:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065589+0000", 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065602+0000", 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2737e-05 2024-04-24T14:32:52.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065620+0000", 2024-04-24T14:32:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8546000000000002e-05 2024-04-24T14:32:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065629+0000", 2024-04-24T14:32:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1600000000000004e-06 2024-04-24T14:32:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065729+0000", 2024-04-24T14:32:52.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9933999999999996e-05 2024-04-24T14:32:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065792+0000", 2024-04-24T14:32:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2712000000000005e-05 2024-04-24T14:32:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068667+0000", 2024-04-24T14:32:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028752920000000002 2024-04-24T14:32:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068790+0000", 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012244400000000001 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:137 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'91)", 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.249264291999999, 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052524399999999997, 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.371 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066945+0000", 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067031+0000", 2024-04-24T14:32:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6228000000000002e-05 2024-04-24T14:32:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067060+0000", 2024-04-24T14:32:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8445999999999998e-05 2024-04-24T14:32:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067069+0000", 2024-04-24T14:32:52.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999996e-06 2024-04-24T14:32:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067172+0000", 2024-04-24T14:32:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010285099999999999 2024-04-24T14:32:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067234+0000", 2024-04-24T14:32:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2068000000000003e-05 2024-04-24T14:32:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072090+0000", 2024-04-24T14:32:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048560859999999999 2024-04-24T14:32:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072199+0000", 2024-04-24T14:32:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108384 2024-04-24T14:32:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:138 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'96, mlcod=13'91)", 2024-04-24T14:32:52.375 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:32:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.248264528, 2024-04-24T14:32:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053159380000000001, 2024-04-24T14:32:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:32:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:32:52.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067945+0000", 2024-04-24T14:32:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067957+0000", 2024-04-24T14:32:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1963e-05 2024-04-24T14:32:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067982+0000", 2024-04-24T14:32:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5570000000000001e-05 2024-04-24T14:32:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067994+0000", 2024-04-24T14:32:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2306e-05 2024-04-24T14:32:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068282+0000", 2024-04-24T14:32:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028799000000000001 2024-04-24T14:32:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068307+0000", 2024-04-24T14:32:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4351000000000001e-05 2024-04-24T14:32:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073142+0000", 2024-04-24T14:32:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004835326 2024-04-24T14:32:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073262+0000", 2024-04-24T14:32:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119695 2024-04-24T14:32:52.379 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:139 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.246781701, 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039447479999999997, 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:32:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069428+0000", 2024-04-24T14:32:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069440+0000", 2024-04-24T14:32:52.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2047e-05 2024-04-24T14:32:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069469+0000", 2024-04-24T14:32:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8852000000000001e-05 2024-04-24T14:32:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069477+0000", 2024-04-24T14:32:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-04-24T14:32:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069531+0000", 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4126999999999999e-05 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069559+0000", 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7881e-05 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073270+0000", 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037104519999999999 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073373+0000", 2024-04-24T14:32:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010380899999999999 2024-04-24T14:32:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:140 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'92)", 2024-04-24T14:32:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:32:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.245955824999999, 2024-04-24T14:32:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031722510000000001, 2024-04-24T14:32:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.384 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:32:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:32:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070253+0000", 2024-04-24T14:32:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070307+0000", 2024-04-24T14:32:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3909000000000002e-05 2024-04-24T14:32:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070330+0000", 2024-04-24T14:32:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2599000000000002e-05 2024-04-24T14:32:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070337+0000", 2024-04-24T14:32:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3030000000000002e-06 2024-04-24T14:32:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070377+0000", 2024-04-24T14:32:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9916000000000002e-05 2024-04-24T14:32:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070398+0000", 2024-04-24T14:32:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0579000000000001e-05 2024-04-24T14:32:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073377+0000", 2024-04-24T14:32:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002979013 2024-04-24T14:32:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073427+0000", 2024-04-24T14:32:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0078999999999999e-05 2024-04-24T14:32:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.388 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:143 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'101, mlcod=13'98)", 2024-04-24T14:32:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:32:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.239770544999999, 2024-04-24T14:32:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030247590000000001, 2024-04-24T14:32:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:32:52.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:32:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076438+0000", 2024-04-24T14:32:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076533+0000", 2024-04-24T14:32:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4443000000000004e-05 2024-04-24T14:32:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076563+0000", 2024-04-24T14:32:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0386999999999999e-05 2024-04-24T14:32:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076604+0000", 2024-04-24T14:32:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1012e-05 2024-04-24T14:32:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076711+0000", 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010637300000000001 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076737+0000", 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6190000000000002e-05 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079368+0000", 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026313529999999999 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079465+0000", 2024-04-24T14:32:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6426000000000001e-05 2024-04-24T14:32:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:159 1.0 e14/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'113, mlcod=14'109)", 2024-04-24T14:32:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:32:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.605950439000001, 2024-04-24T14:32:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052053589999999997, 2024-04-24T14:32:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.393 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:32:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:32:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710259+0000", 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710272+0000", 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.382e-05 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710303+0000", 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0629000000000001e-05 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710322+0000", 2024-04-24T14:32:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8859999999999999e-05 2024-04-24T14:32:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710479+0000", 2024-04-24T14:32:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157076 2024-04-24T14:32:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710535+0000", 2024-04-24T14:32:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6289000000000003e-05 2024-04-24T14:32:52.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715391+0000", 2024-04-24T14:32:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048557139999999997 2024-04-24T14:32:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715465+0000", 2024-04-24T14:32:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4301000000000006e-05 2024-04-24T14:32:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:161 1.0 e14/12 1:30306672:devicehealth::main.db.0000000000000000:head v 14'115, mlcod=14'112)", 2024-04-24T14:32:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:32:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.598839637000001, 2024-04-24T14:32:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039960450000000002, 2024-04-24T14:32:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:32:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:32:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717370+0000", 2024-04-24T14:32:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717499+0000", 2024-04-24T14:32:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012877400000000001 2024-04-24T14:32:52.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717513+0000", 2024-04-24T14:32:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4229999999999999e-05 2024-04-24T14:32:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717521+0000", 2024-04-24T14:32:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2024-04-24T14:32:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718040+0000", 2024-04-24T14:32:52.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051920099999999995 2024-04-24T14:32:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718094+0000", 2024-04-24T14:32:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859000000000001e-05 2024-04-24T14:32:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721278+0000", 2024-04-24T14:32:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031838560000000001 2024-04-24T14:32:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:52.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721367+0000", 2024-04-24T14:32:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9241999999999997e-05 2024-04-24T14:32:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:52.402 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:52.402 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:52.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:32:52.404 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:52.407+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:52.406 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:32:52.430 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:32:52.451 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:32:52.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T14:32:52.472 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:32:52.500 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:32:52.503 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:52.507+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:52.512 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:52.512 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:52.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:32:52.525 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:32:52.527 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:32:52.527 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:32:52.527 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:32:52.527 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:32:52.527 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:32:52.527 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:32:52.605 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:52.607+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:52.616 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:52.616 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:52.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:32:52.706 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:52.711+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:52.715 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:52.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:32:52.807 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:52.807+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:52.819 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:52.819 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:52.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:32:52.908 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:52.911+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:52.925 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:52.925 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:52.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:32:53.009 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:53.015+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:53.081 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:53.081 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:53.090 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:32:53.109 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:53.111+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:53.200 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:53.210 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:32:53.211 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:53.215+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:53.311 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:53.311 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:53.315+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:53.320 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:32:53.411 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:53.415+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:53.415 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:53.415 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:53.424 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:32:53.456 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-24T14:32:53.513 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:53.515+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:53.519 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:53.519 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:53.528 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:32:53.614 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:53.615+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:53.621 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:53.621 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:53.630 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:32:53.715 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:53.715+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:53.773 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:53.773 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:53.782 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:32:53.815 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:53.819+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:53.883 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:53.884 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:53.892 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:32:53.935 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:53.939+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:54.017 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:54.023+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:54.036 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:54.036 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:54.045 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:32:54.118 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:54.119+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:54.143 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:54.143 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:54.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:32:54.219 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:54.223+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:54.260 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:54.260 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:54.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:32:54.320 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:54.323+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:54.370 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:54.371 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:54.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:32:54.421 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:54.423+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:54.478 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:54.478 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:54.487 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:32:54.523 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:54.523+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:54.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:32:54.582 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:54.583 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:54.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:32:54.635 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:54.639+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:54.669 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:32:54.687 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:32:54.707 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:32:54.724 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:54.727+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:54.732 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:32:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:32:54.732 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:54.735 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:32:54.740 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:32:54.756 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:32:54.782 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:32:54.784 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:32:54.784 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:32:54.785 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:32:54.785 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:32:54.785 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:32:54.785 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:32:54.832 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:54.833 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:32:54.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:32:54.833 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-04-24T14:32:54.833 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:54.833 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:54.835+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:54.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:32:54.925 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:54.931+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:54.985 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:32:54.985 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:54.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:32:55.026 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:55.027+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:55.095 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:55.095 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:32:55.096 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:32:55.096 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:55.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:32:55.127 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:55.131+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:55.185 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T14:32:55.185 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:32:55.185 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-04-24T14:32:55.185 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T14:32:55.186 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T14:32:55.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-04-24T14:32:55.227 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:55.231+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:55.249 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:32:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:32:55.249 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-04-24T14:32:55.249 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:55.259 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:32:55.288 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:55.288 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-24T14:32:55.288 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:55.328 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:55.331+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:55.368 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:55.368 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:55.368 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:32:55.368 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:55.377 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:32:55.429 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:55.431+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:55.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T14:32:55.477 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:55.477 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:55.477 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:32:55.477 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:32:55.477 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:55.487 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:32:55.530 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:55.531+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:55.630 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:55.631+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:55.639 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:55.639 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:32:55.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:32:55.639 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:32:55.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.639 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.639 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:32:55.639 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 33.315677891, 2024-04-24T14:32:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011830553000000001, 2024-04-24T14:32:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 36 2024-04-24T14:32:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:32:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:32:55.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326000+0000", 2024-04-24T14:32:55.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:55.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326006+0000", 2024-04-24T14:32:55.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9340000000000003e-06 2024-04-24T14:32:55.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326044+0000", 2024-04-24T14:32:55.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8124999999999998e-05 2024-04-24T14:32:55.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326054+0000", 2024-04-24T14:32:55.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9650000000000003e-06 2024-04-24T14:32:55.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332746+0000", 2024-04-24T14:32:55.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006692539 2024-04-24T14:32:55.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332833+0000", 2024-04-24T14:32:55.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6482000000000003e-05 2024-04-24T14:32:55.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332947+0000", 2024-04-24T14:32:55.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114472 2024-04-24T14:32:55.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334320+0000", 2024-04-24T14:32:55.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001372489 2024-04-24T14:32:55.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337717+0000", 2024-04-24T14:32:55.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003396849 2024-04-24T14:32:55.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337789+0000", 2024-04-24T14:32:55.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2080000000000001e-05 2024-04-24T14:32:55.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337831+0000", 2024-04-24T14:32:55.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.244e-05 2024-04-24T14:32:55.645 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.645 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.645 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.646 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.646 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:32:55.646 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 33.315616028999997, 2024-04-24T14:32:55.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:32:55.646 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.646 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.646 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.646 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.646 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.646 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:32:55.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.646 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:32:55.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:32:55.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:32:55.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:32:55.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:32:55.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:32:55.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:32:55.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:32:55.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:32:55.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:32:55.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:32:55.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:32:55.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:32:55.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:32:55.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:32:55.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:32:55.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:32:55.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:32:55.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:32:55.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:32:55.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:32:55.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:32:55.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:32:55.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:32:55.651 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.651 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.652 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.652 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.652 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:32:55.652 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 33.222111556000002, 2024-04-24T14:32:55.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011218514000000001, 2024-04-24T14:32:55.652 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 88 2024-04-24T14:32:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:32:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:32:55.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419565+0000", 2024-04-24T14:32:55.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:55.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419575+0000", 2024-04-24T14:32:55.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5340000000000005e-06 2024-04-24T14:32:55.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419616+0000", 2024-04-24T14:32:55.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1301999999999999e-05 2024-04-24T14:32:55.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419638+0000", 2024-04-24T14:32:55.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2177000000000001e-05 2024-04-24T14:32:55.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428578+0000", 2024-04-24T14:32:55.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089398280000000004 2024-04-24T14:32:55.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428665+0000", 2024-04-24T14:32:55.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6644000000000006e-05 2024-04-24T14:32:55.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428797+0000", 2024-04-24T14:32:55.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013231500000000001 2024-04-24T14:32:55.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.429866+0000", 2024-04-24T14:32:55.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001069221 2024-04-24T14:32:55.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430684+0000", 2024-04-24T14:32:55.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081788399999999995 2024-04-24T14:32:55.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430743+0000", 2024-04-24T14:32:55.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8875999999999997e-05 2024-04-24T14:32:55.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430786+0000", 2024-04-24T14:32:55.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2398999999999998e-05 2024-04-24T14:32:55.658 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.658 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.658 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.658 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.658 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:32:55.658 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 33.016101087000003, 2024-04-24T14:32:55.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:32:55.658 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.659 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.659 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.659 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.659 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.659 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:32:55.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.659 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:32:55.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.660 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:32:55.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.660 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.660 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:32:55.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:55.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.660 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.660 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:32:55.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:32:55.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:32:55.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:32:55.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:32:55.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:32:55.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:32:55.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:32:55.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:32:55.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:32:55.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:32:55.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:32:55.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:32:55.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:32:55.664 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.664 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.664 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.664 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.664 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:32:55.664 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 33.015875319000003, 2024-04-24T14:32:55.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:32:55.664 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.664 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.664 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:32:55.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:55.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:32:55.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:32:55.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:32:55.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:32:55.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:32:55.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:32:55.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:32:55.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:32:55.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:32:55.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:32:55.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:32:55.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:32:55.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:32:55.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:32:55.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:32:55.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:32:55.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:32:55.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:32:55.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:32:55.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:32:55.670 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.670 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.670 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.670 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.670 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:32:55.670 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 33.015698231999998, 2024-04-24T14:32:55.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:32:55.670 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.670 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.670 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.670 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.671 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.671 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:32:55.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.671 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:32:55.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:32:55.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:32:55.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:55.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:32:55.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:32:55.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:32:55.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:32:55.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:32:55.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:32:55.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:32:55.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:32:55.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:32:55.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:32:55.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:32:55.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:32:55.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:32:55.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:32:55.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:32:55.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:32:55.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:32:55.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:32:55.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:32:55.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:32:55.676 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.676 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.676 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.676 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.676 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:32:55.676 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 33.015594317999998, 2024-04-24T14:32:55.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:32:55.676 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.676 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.676 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.677 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.677 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.677 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:32:55.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.677 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:32:55.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:32:55.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:32:55.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:32:55.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:32:55.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:32:55.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:32:55.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:32:55.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:32:55.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:32:55.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:32:55.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:32:55.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:32:55.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:32:55.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:32:55.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:32:55.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:32:55.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:32:55.682 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.682 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.682 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.682 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.682 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:32:55.682 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 33.015297998000001, 2024-04-24T14:32:55.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:32:55.682 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.682 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.682 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.683 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.683 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.683 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:32:55.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.683 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.683 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:32:55.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:32:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:32:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:32:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:32:55.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:32:55.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:32:55.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:32:55.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:32:55.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:32:55.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:32:55.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:32:55.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:32:55.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:32:55.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:32:55.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:32:55.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:32:55.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:32:55.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:32:55.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:32:55.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:32:55.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:32:55.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:32:55.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:32:55.688 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.688 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.688 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.688 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.689 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:32:55.689 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 33.015205866999999, 2024-04-24T14:32:55.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:32:55.689 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.689 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.689 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.689 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.689 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.689 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:32:55.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.689 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.689 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:32:55.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:32:55.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:32:55.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:55.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:32:55.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:32:55.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:32:55.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:32:55.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:32:55.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:32:55.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:32:55.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:32:55.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:32:55.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:32:55.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:32:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:32:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:32:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:32:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 33.015120654999997, 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.695 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:32:55.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:32:55.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:32:55.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:55.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:32:55.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:32:55.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:32:55.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:32:55.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:32:55.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:32:55.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.698 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:32:55.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:32:55.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.698 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:32:55.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:32:55.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:32:55.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:32:55.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:32:55.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:32:55.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:32:55.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:32:55.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.700 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.700 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:32:55.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:32:55.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.700 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.700 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:32:55.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:32:55.700 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.700 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.700 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.701 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.701 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:32:55.701 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 33.014883773999998, 2024-04-24T14:32:55.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:32:55.701 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.701 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.701 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.701 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.701 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.701 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:32:55.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.701 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.702 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.702 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:32:55.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.702 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.702 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:32:55.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.702 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.702 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:32:55.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:32:55.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:32:55.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:32:55.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:32:55.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:32:55.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:32:55.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:32:55.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.704 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:32:55.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:32:55.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.704 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:32:55.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:32:55.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:32:55.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:32:55.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:32:55.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:32:55.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:32:55.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:32:55.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:32:55.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:32:55.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:32:55.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:32:55.706 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.706 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.706 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.707 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.707 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:32:55.707 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 33.014802252000003, 2024-04-24T14:32:55.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:32:55.707 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.707 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.707 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.707 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.707 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.707 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:32:55.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.707 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:32:55.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:32:55.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:32:55.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:55.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:32:55.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:32:55.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:32:55.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:32:55.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:32:55.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:32:55.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:32:55.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:32:55.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:32:55.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:32:55.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:32:55.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:32:55.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:32:55.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:32:55.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:32:55.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:32:55.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:32:55.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:32:55.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:32:55.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:32:55.712 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.712 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.712 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.713 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.713 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:32:55.713 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.583083176000002, 2024-04-24T14:32:55.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01155111, 2024-04-24T14:32:55.713 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.713 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.713 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.713 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.713 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.713 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135 2024-04-24T14:32:55.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.714 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.714 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.714 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.714 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:32:55.714 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.714 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.714 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.714 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.714 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:32:55.714 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.714 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.714 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.714 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.715 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:32:55.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:55.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.715 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058602+0000", 2024-04-24T14:32:55.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1239999999999999e-06 2024-04-24T14:32:55.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.715 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058609+0000", 2024-04-24T14:32:55.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6719999999999998e-06 2024-04-24T14:32:55.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058626+0000", 2024-04-24T14:32:55.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7569999999999999e-05 2024-04-24T14:32:55.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065271+0000", 2024-04-24T14:32:55.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00664469 2024-04-24T14:32:55.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065398+0000", 2024-04-24T14:32:55.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012694299999999999 2024-04-24T14:32:55.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065598+0000", 2024-04-24T14:32:55.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000200216 2024-04-24T14:32:55.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067596+0000", 2024-04-24T14:32:55.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019976450000000001 2024-04-24T14:32:55.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070055+0000", 2024-04-24T14:32:55.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024593710000000001 2024-04-24T14:32:55.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.718 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070109+0000", 2024-04-24T14:32:55.718 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3690999999999998e-05 2024-04-24T14:32:55.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.718 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070147+0000", 2024-04-24T14:32:55.718 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7910000000000001e-05 2024-04-24T14:32:55.718 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.718 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.583043840999999, 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.719 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.720 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.720 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.720 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:32:55.720 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.720 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.720 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.720 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.720 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:32:55.720 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.720 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.720 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.720 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.720 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:32:55.720 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:32:55.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.721 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.721 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:32:55.721 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:32:55.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.721 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.721 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:32:55.721 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:32:55.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.721 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.721 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:32:55.722 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:32:55.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.722 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.722 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:32:55.722 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:32:55.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.722 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.722 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:32:55.722 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:32:55.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.724 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.724 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:32:55.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:32:55.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.724 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.724 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.724 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:32:55.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:32:55.724 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.724 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.724 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.724 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.724 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.583002338, 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:32:55.725 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.726 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:32:55.726 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.726 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:32:55.726 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:32:55.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.726 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:32:55.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:32:55.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.727 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.727 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.727 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:32:55.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:32:55.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.727 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.727 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.727 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:32:55.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:32:55.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.727 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.727 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.728 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:32:55.728 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:32:55.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.728 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.728 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.728 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:32:55.728 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:32:55.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.728 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.728 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.728 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:32:55.728 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:32:55.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.728 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.729 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:32:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:32:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:32:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.582889905000002, 2024-04-24T14:32:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:32:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.730 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.731 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.731 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.731 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.731 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:32:55.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.731 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.731 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.731 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:32:55.731 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.731 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.731 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:32:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:32:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:32:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:32:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.733 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.734 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:32:55.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:32:55.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.734 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:32:55.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:32:55.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.734 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:32:55.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:32:55.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.735 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.735 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:32:55.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:32:55.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.735 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.735 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:32:55.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:32:55.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.735 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.735 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:32:55.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:32:55.736 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.736 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.736 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.736 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.736 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.736 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.736 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:32:55.736 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.582861604999998, 2024-04-24T14:32:55.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:32:55.736 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.736 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.736 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.736 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.737 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.737 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:32:55.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.737 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.737 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.737 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:32:55.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.737 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.737 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:32:55.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:32:55.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:32:55.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:32:55.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:32:55.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:32:55.739 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:32:55.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.739 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.739 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:32:55.739 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:32:55.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.739 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.739 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:32:55.739 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:32:55.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.740 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.740 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:32:55.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:32:55.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.740 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.740 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.740 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:32:55.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:32:55.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.740 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.740 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.740 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:32:55.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:32:55.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.741 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:32:55.741 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:32:55.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.741 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:32:55.741 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:32:55.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.741 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:32:55.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:32:55.742 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.742 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.742 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.742 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.742 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:32:55.742 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.582773901000003, 2024-04-24T14:32:55.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:32:55.742 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.742 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.742 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.742 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.743 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.743 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:32:55.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.743 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:32:55.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:32:55.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.744 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.744 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:32:55.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:55.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.744 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.744 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:32:55.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:32:55.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.744 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.744 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:32:55.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:32:55.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:32:55.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:32:55.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:32:55.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:32:55.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:32:55.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:32:55.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:32:55.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:32:55.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:32:55.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:32:55.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:32:55.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:32:55.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:32:55.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:32:55.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:32:55.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:32:55.747 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.748 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.748 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.748 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:32:55.748 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:32:55.748 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.5826511, 2024-04-24T14:32:55.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:32:55.748 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.748 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.748 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.748 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.748 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.748 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:32:55.749 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.749 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.749 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.749 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:32:55.749 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.749 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.749 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.749 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:32:55.749 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.749 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.749 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.750 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:32:55.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:32:55.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.750 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.750 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.750 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:32:55.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:32:55.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.750 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.750 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.750 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:32:55.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:32:55.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.750 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:32:55.751 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:32:55.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:32:55.751 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:32:55.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:32:55.751 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:32:55.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:32:55.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.753 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.753 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.753 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:32:55.753 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:32:55.753 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.753 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.753 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.753 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:32:55.753 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:32:55.753 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.753 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.753 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.753 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.754 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.754 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:32:55.754 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:32:55.754 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.939411167999999, 2024-04-24T14:32:55.754 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:32:55.754 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:32:55.754 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:55.754 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:32:55.754 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:32:55.754 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:32:55.754 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:32:55.754 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.754 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:32:55.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.756 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.756 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:32:55.756 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:32:55.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:32:55.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.756 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.756 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:32:55.756 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:32:55.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:32:55.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.756 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.756 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:32:55.756 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:32:55.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:32:55.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:32:55.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.758 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.758 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:32:55.758 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:32:55.758 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:32:55.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.758 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.758 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:32:55.758 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:32:55.758 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:32:55.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.758 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.759 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:32:55.759 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:32:55.759 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:32:55.759 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:32:55.759 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:32:55.759 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:32:55.759 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:32:55.759 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:32:55.759 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.759 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.759 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.759 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:32:55.759 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:32:55.759 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:55.761 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:32:55.761 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:55.767+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:55.862 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:55.867+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:55.868 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:55.868 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:32:55.868 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:55.878 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:32:55.963 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:55.967+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:55.973 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:55.974 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:55.974 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:32:55.974 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:32:55.974 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:55.982 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:32:56.064 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:56.067+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:56.080 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:56.080 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:32:56.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:32:56.080 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-24T14:32:56.080 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:56.089 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:32:56.164 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:56.167+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:56.234 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:56.234 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:32:56.234 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:56.243 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:32:56.265 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:56.267+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:56.343 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:56.343 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:56.344 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:32:56.344 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:32:56.344 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:56.352 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:32:56.395 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:56.399+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:56.466 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:56.467+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:56.472 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-24T14:32:56.496 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:56.496 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:32:56.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:32:56.496 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-24T14:32:56.496 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:56.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:32:56.567 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:56.571+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:56.635 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:56.636 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:56.636 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:32:56.636 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:56.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:32:56.668 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:56.671+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:56.745 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:56.745 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:56.745 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:32:56.745 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:32:56.745 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:56.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:32:56.769 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:56.771+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:56.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:32:56.859 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:56.859 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:32:56.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:32:56.859 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:32:56.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.859 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:36 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'22, mlcod=13'17)", 2024-04-24T14:32:56.859 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:32:56.859 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.530380921000003, 2024-04-24T14:32:56.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043791400000000001, 2024-04-24T14:32:56.859 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.859 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.859 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:32:56.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:32:56.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331851+0000", 2024-04-24T14:32:56.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331864+0000", 2024-04-24T14:32:56.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2767e-05 2024-04-24T14:32:56.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331881+0000", 2024-04-24T14:32:56.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6586e-05 2024-04-24T14:32:56.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331894+0000", 2024-04-24T14:32:56.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2976000000000001e-05 2024-04-24T14:32:56.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332161+0000", 2024-04-24T14:32:56.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026765399999999998 2024-04-24T14:32:56.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332197+0000", 2024-04-24T14:32:56.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5565999999999998e-05 2024-04-24T14:32:56.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336158+0000", 2024-04-24T14:32:56.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039616820000000002 2024-04-24T14:32:56.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336232+0000", 2024-04-24T14:32:56.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3151999999999994e-05 2024-04-24T14:32:56.863 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.863 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.863 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.864 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:37 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'23, mlcod=13'18)", 2024-04-24T14:32:56.864 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:32:56.864 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.529494442999997, 2024-04-24T14:32:56.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040417389999999999, 2024-04-24T14:32:56.864 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.864 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.864 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:32:56.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:32:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332738+0000", 2024-04-24T14:32:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332750+0000", 2024-04-24T14:32:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2432000000000001e-05 2024-04-24T14:32:56.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332768+0000", 2024-04-24T14:32:56.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7314000000000001e-05 2024-04-24T14:32:56.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332780+0000", 2024-04-24T14:32:56.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2716e-05 2024-04-24T14:32:56.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332925+0000", 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144587 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332955+0000", 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0246000000000001e-05 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336698+0000", 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037430559999999998 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336781+0000", 2024-04-24T14:32:56.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2447999999999994e-05 2024-04-24T14:32:56.868 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.868 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.868 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.868 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:39 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'25, mlcod=13'22)", 2024-04-24T14:32:56.868 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:32:56.868 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.523376120000002, 2024-04-24T14:32:56.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038736740000000001, 2024-04-24T14:32:56.868 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.868 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.868 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338856+0000", 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338995+0000", 2024-04-24T14:32:56.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013868100000000001 2024-04-24T14:32:56.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339029+0000", 2024-04-24T14:32:56.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4372000000000002e-05 2024-04-24T14:32:56.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339050+0000", 2024-04-24T14:32:56.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0656999999999999e-05 2024-04-24T14:32:56.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339156+0000", 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106177 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339188+0000", 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.243e-05 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342648+0000", 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034601689999999999 2024-04-24T14:32:56.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342731+0000", 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2534999999999995e-05 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:67 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'41, mlcod=13'39)", 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.472639835999999, 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031029009999999999, 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.872 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:32:56.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:32:56.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389592+0000", 2024-04-24T14:32:56.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389603+0000", 2024-04-24T14:32:56.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090000000000001e-05 2024-04-24T14:32:56.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389631+0000", 2024-04-24T14:32:56.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7772000000000001e-05 2024-04-24T14:32:56.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389646+0000", 2024-04-24T14:32:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.432e-05 2024-04-24T14:32:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389732+0000", 2024-04-24T14:32:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6331999999999999e-05 2024-04-24T14:32:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389771+0000", 2024-04-24T14:32:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8751999999999998e-05 2024-04-24T14:32:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392585+0000", 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028147469999999998 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392696+0000", 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111121 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:32:56.876 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.471497225, 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030475519999999998, 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390735+0000", 2024-04-24T14:32:56.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390804+0000", 2024-04-24T14:32:56.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9488999999999998e-05 2024-04-24T14:32:56.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390832+0000", 2024-04-24T14:32:56.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7946e-05 2024-04-24T14:32:56.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390847+0000", 2024-04-24T14:32:56.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4693e-05 2024-04-24T14:32:56.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391508+0000", 2024-04-24T14:32:56.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066154799999999998 2024-04-24T14:32:56.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391538+0000", 2024-04-24T14:32:56.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0029e-05 2024-04-24T14:32:56.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393698+0000", 2024-04-24T14:32:56.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002160052 2024-04-24T14:32:56.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393784+0000", 2024-04-24T14:32:56.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5414999999999995e-05 2024-04-24T14:32:56.880 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.880 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.880 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.881 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:72 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'46, mlcod=13'43)", 2024-04-24T14:32:56.881 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:32:56.881 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.465323732000002, 2024-04-24T14:32:56.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036059460000000001, 2024-04-24T14:32:56.881 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.881 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.881 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:32:56.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:32:56.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396908+0000", 2024-04-24T14:32:56.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396968+0000", 2024-04-24T14:32:56.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9412e-05 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397010+0000", 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2169000000000002e-05 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397022+0000", 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1749999999999999e-05 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397096+0000", 2024-04-24T14:32:56.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4701000000000002e-05 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397126+0000", 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400443+0000", 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033172140000000002 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400516+0000", 2024-04-24T14:32:56.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2181999999999994e-05 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:73 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'47, mlcod=13'43)", 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.464476328000003, 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055152170000000002, 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:32:56.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:32:56.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397756+0000", 2024-04-24T14:32:56.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397884+0000", 2024-04-24T14:32:56.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012842100000000001 2024-04-24T14:32:56.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397923+0000", 2024-04-24T14:32:56.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8942000000000001e-05 2024-04-24T14:32:56.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397939+0000", 2024-04-24T14:32:56.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6073e-05 2024-04-24T14:32:56.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398106+0000", 2024-04-24T14:32:56.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016737100000000001 2024-04-24T14:32:56.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398137+0000", 2024-04-24T14:32:56.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0496000000000001e-05 2024-04-24T14:32:56.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403191+0000", 2024-04-24T14:32:56.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00505391 2024-04-24T14:32:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403272+0000", 2024-04-24T14:32:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1483999999999994e-05 2024-04-24T14:32:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:74 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'48, mlcod=13'43)", 2024-04-24T14:32:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:32:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.463677302000001, 2024-04-24T14:32:56.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048091760000000001, 2024-04-24T14:32:56.890 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.890 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.890 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:32:56.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:32:56.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398555+0000", 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398570+0000", 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.558e-05 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398610+0000", 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0028999999999999e-05 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398625+0000", 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398701+0000", 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6507000000000005e-05 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398731+0000", 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9356000000000002e-05 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403281+0000", 2024-04-24T14:32:56.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004549796 2024-04-24T14:32:56.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403365+0000", 2024-04-24T14:32:56.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4685000000000007e-05 2024-04-24T14:32:56.893 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.893 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.893 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.442364376999997, 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038140460000000002, 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419868+0000", 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419882+0000", 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.45e-05 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419919+0000", 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6939e-05 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419961+0000", 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2148999999999999e-05 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420084+0000", 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122958 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420122+0000", 2024-04-24T14:32:56.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8334999999999998e-05 2024-04-24T14:32:56.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423616+0000", 2024-04-24T14:32:56.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003493566 2024-04-24T14:32:56.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423683+0000", 2024-04-24T14:32:56.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7149000000000004e-05 2024-04-24T14:32:56.897 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.898 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.898 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.898 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T14:32:56.898 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:32:56.898 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.440952621999998, 2024-04-24T14:32:56.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030290730000000002, 2024-04-24T14:32:56.898 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.898 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.898 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:32:56.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:32:56.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421279+0000", 2024-04-24T14:32:56.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421352+0000", 2024-04-24T14:32:56.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2251999999999999e-05 2024-04-24T14:32:56.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421378+0000", 2024-04-24T14:32:56.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6006000000000001e-05 2024-04-24T14:32:56.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421388+0000", 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0373e-05 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421473+0000", 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4665000000000003e-05 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421512+0000", 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9619000000000001e-05 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424198+0000", 2024-04-24T14:32:56.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026854610000000001 2024-04-24T14:32:56.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424310+0000", 2024-04-24T14:32:56.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011208 2024-04-24T14:32:56.902 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.902 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.902 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.902 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:87 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'57, mlcod=13'51)", 2024-04-24T14:32:56.902 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:32:56.902 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.436689663000003, 2024-04-24T14:32:56.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002974057, 2024-04-24T14:32:56.903 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.903 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.903 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:32:56.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:32:56.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425542+0000", 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425614+0000", 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1328999999999994e-05 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425664+0000", 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0491999999999997e-05 2024-04-24T14:32:56.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425676+0000", 2024-04-24T14:32:56.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.134e-05 2024-04-24T14:32:56.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425798+0000", 2024-04-24T14:32:56.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012221700000000001 2024-04-24T14:32:56.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425860+0000", 2024-04-24T14:32:56.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1761999999999996e-05 2024-04-24T14:32:56.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428368+0000", 2024-04-24T14:32:56.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025084 2024-04-24T14:32:56.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428518+0000", 2024-04-24T14:32:56.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149884 2024-04-24T14:32:56.906 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.906 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.906 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.236580938000003, 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002919694, 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:32:56.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625651+0000", 2024-04-24T14:32:56.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:56.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625667+0000", 2024-04-24T14:32:56.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5973000000000001e-05 2024-04-24T14:32:56.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625689+0000", 2024-04-24T14:32:56.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:32:56.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625705+0000", 2024-04-24T14:32:56.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5797000000000001e-05 2024-04-24T14:32:56.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625805+0000", 2024-04-24T14:32:56.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9238000000000001e-05 2024-04-24T14:32:56.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625837+0000", 2024-04-24T14:32:56.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2722000000000003e-05 2024-04-24T14:32:56.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628476+0000", 2024-04-24T14:32:56.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026391980000000002 2024-04-24T14:32:56.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628572+0000", 2024-04-24T14:32:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5754000000000005e-05 2024-04-24T14:32:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'88)", 2024-04-24T14:32:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:32:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.796643060000001, 2024-04-24T14:32:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031994179999999999, 2024-04-24T14:32:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.911 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:32:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:32:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065589+0000", 2024-04-24T14:32:56.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065602+0000", 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2737e-05 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065620+0000", 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8546000000000002e-05 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065629+0000", 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1600000000000004e-06 2024-04-24T14:32:56.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065729+0000", 2024-04-24T14:32:56.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9933999999999996e-05 2024-04-24T14:32:56.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065792+0000", 2024-04-24T14:32:56.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2712000000000005e-05 2024-04-24T14:32:56.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068667+0000", 2024-04-24T14:32:56.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028752920000000002 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068790+0000", 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012244400000000001 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:137 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'91)", 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.795287324, 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052524399999999997, 2024-04-24T14:32:56.915 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066945+0000", 2024-04-24T14:32:56.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067031+0000", 2024-04-24T14:32:56.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6228000000000002e-05 2024-04-24T14:32:56.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067060+0000", 2024-04-24T14:32:56.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8445999999999998e-05 2024-04-24T14:32:56.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067069+0000", 2024-04-24T14:32:56.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999996e-06 2024-04-24T14:32:56.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067172+0000", 2024-04-24T14:32:56.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010285099999999999 2024-04-24T14:32:56.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067234+0000", 2024-04-24T14:32:56.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2068000000000003e-05 2024-04-24T14:32:56.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072090+0000", 2024-04-24T14:32:56.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048560859999999999 2024-04-24T14:32:56.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072199+0000", 2024-04-24T14:32:56.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108384 2024-04-24T14:32:56.919 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.919 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:138 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'96, mlcod=13'91)", 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.794287560000001, 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053159380000000001, 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:32:56.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067945+0000", 2024-04-24T14:32:56.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067957+0000", 2024-04-24T14:32:56.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1963e-05 2024-04-24T14:32:56.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067982+0000", 2024-04-24T14:32:56.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5570000000000001e-05 2024-04-24T14:32:56.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067994+0000", 2024-04-24T14:32:56.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2306e-05 2024-04-24T14:32:56.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068282+0000", 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028799000000000001 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068307+0000", 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4351000000000001e-05 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073142+0000", 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004835326 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073262+0000", 2024-04-24T14:32:56.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119695 2024-04-24T14:32:56.924 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.924 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.924 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.924 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:139 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T14:32:56.924 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:32:56.924 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.792804732999997, 2024-04-24T14:32:56.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039447479999999997, 2024-04-24T14:32:56.924 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.924 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.924 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069428+0000", 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069440+0000", 2024-04-24T14:32:56.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2047e-05 2024-04-24T14:32:56.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069469+0000", 2024-04-24T14:32:56.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8852000000000001e-05 2024-04-24T14:32:56.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069477+0000", 2024-04-24T14:32:56.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-04-24T14:32:56.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069531+0000", 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4126999999999999e-05 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069559+0000", 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7881e-05 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073270+0000", 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037104519999999999 2024-04-24T14:32:56.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073373+0000", 2024-04-24T14:32:56.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010380899999999999 2024-04-24T14:32:56.928 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.928 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.928 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.928 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:140 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'92)", 2024-04-24T14:32:56.928 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:32:56.928 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.791978856999997, 2024-04-24T14:32:56.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031722510000000001, 2024-04-24T14:32:56.928 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.929 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.929 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:32:56.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:32:56.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070253+0000", 2024-04-24T14:32:56.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:56.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070307+0000", 2024-04-24T14:32:56.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3909000000000002e-05 2024-04-24T14:32:56.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070330+0000", 2024-04-24T14:32:56.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2599000000000002e-05 2024-04-24T14:32:56.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070337+0000", 2024-04-24T14:32:56.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3030000000000002e-06 2024-04-24T14:32:56.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070377+0000", 2024-04-24T14:32:56.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9916000000000002e-05 2024-04-24T14:32:56.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070398+0000", 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0579000000000001e-05 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073377+0000", 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002979013 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073427+0000", 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0078999999999999e-05 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.933 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:143 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'101, mlcod=13'98)", 2024-04-24T14:32:56.933 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:32:56.933 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.785793577, 2024-04-24T14:32:56.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030247590000000001, 2024-04-24T14:32:56.933 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.933 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.933 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:32:56.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:32:56.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076438+0000", 2024-04-24T14:32:56.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076533+0000", 2024-04-24T14:32:56.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4443000000000004e-05 2024-04-24T14:32:56.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076563+0000", 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0386999999999999e-05 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076604+0000", 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1012e-05 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076711+0000", 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010637300000000001 2024-04-24T14:32:56.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076737+0000", 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6190000000000002e-05 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079368+0000", 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026313529999999999 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079465+0000", 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6426000000000001e-05 2024-04-24T14:32:56.936 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:159 1.0 e14/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'113, mlcod=14'109)", 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.151973470999998, 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052053589999999997, 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:32:56.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:32:56.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710259+0000", 2024-04-24T14:32:56.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:32:56.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710272+0000", 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.382e-05 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710303+0000", 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0629000000000001e-05 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710322+0000", 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8859999999999999e-05 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710479+0000", 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157076 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710535+0000", 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6289000000000003e-05 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715391+0000", 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048557139999999997 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715465+0000", 2024-04-24T14:32:56.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4301000000000006e-05 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:161 1.0 e14/12 1:30306672:devicehealth::main.db.0000000000000000:head v 14'115, mlcod=14'112)", 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.144862668999998, 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039960450000000002, 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:32:56.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:32:56.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:32:56.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:32:56.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:32:56.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:32:56.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717370+0000", 2024-04-24T14:32:56.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:32:56.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:32:56.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717499+0000", 2024-04-24T14:32:56.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012877400000000001 2024-04-24T14:32:56.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:32:56.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717513+0000", 2024-04-24T14:32:56.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4229999999999999e-05 2024-04-24T14:32:56.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:32:56.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717521+0000", 2024-04-24T14:32:56.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2024-04-24T14:32:56.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718040+0000", 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051920099999999995 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718094+0000", 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859000000000001e-05 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721278+0000", 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031838560000000001 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:32:56.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:32:56.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721367+0000", 2024-04-24T14:32:56.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9241999999999997e-05 2024-04-24T14:32:56.945 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.945 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.945 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.945 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:32:56.945 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:32:56.945 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:56.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:32:56.947 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:32:56.947 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:56.947+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:56.948 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:32:56.967 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:32:56.995 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:32:57.016 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:32:57.039 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:32:57.041 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:32:57.041 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:32:57.042 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:32:57.042 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:32:57.042 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:32:57.042 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:32:57.044 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:57.044 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:57.047 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:57.051+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:57.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:32:57.149 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:57.151+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:57.200 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:57.200 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:57.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:32:57.250 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:57.251+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:57.310 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:57.310 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:57.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:32:57.351 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:57.355+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:57.419 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:57.419 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:57.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:32:57.451 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:57.455+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:57.524 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:57.524 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:57.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:32:57.552 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:57.555+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:57.631 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:57.631 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:57.640 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:32:57.653 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:57.655+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:57.749 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:57.749 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:57.753 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:57.755+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:57.759 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:32:57.855 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:57.855+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:57.855 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:57.855 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:57.865 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:32:57.955 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:57.959+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:57.963 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:57.963 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:57.972 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:32:58.056 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:58.059+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:58.072 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:58.072 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:58.081 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:32:58.157 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:58.159+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:58.224 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:58.224 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:58.233 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:32:58.258 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:58.259+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:58.328 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:58.328 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:58.337 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:32:58.359 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:58.363+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:58.433 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:58.433 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:58.442 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:32:58.460 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:58.463+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:58.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T14:32:58.544 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:58.544 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:58.553 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:32:58.595 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:58.599+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:58.661 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:58.663+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:58.696 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:58.696 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:58.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:32:58.763 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:58.767+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:58.806 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:58.806 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:58.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:32:58.863 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:58.867+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:58.910 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:58.910 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:58.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:32:58.964 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:32:58.967+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:32:59.022 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:59.022 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:59.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:32:59.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:32:59.066 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:59.067+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:32:59.129 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:32:59.129 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:32:59.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:32:59.166 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:59.167+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:59.176 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:32:59.193 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:32:59.212 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:32:59.232 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:59.232 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:59.233 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:32:59.233 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:32:59.233 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:59.241 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:32:59.267 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:59.271+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:59.333 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:59.333 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:32:59.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:32:59.333 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-04-24T14:32:59.333 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:59.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:32:59.368 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:59.371+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:59.449 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:32:59.469 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:59.471+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:59.485 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-24T14:32:59.485 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:32:59.507 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:32:59.508 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:32:59.509 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:32:59.509 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:32:59.509 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:32:59.509 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:32:59.509 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:32:59.513 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:59.514 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:59.514 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:32:59.514 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:59.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:32:59.569 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:32:59.571+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:32:59.617 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:32:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:32:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:32:59.617 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:59.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:32:59.670 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:32:59.671+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:32:59.730 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:32:59.730 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:32:59.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:32:59.730 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-04-24T14:32:59.730 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:32:59.740 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:32:59.783 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:32:59.787+0000 7f2f442f6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 20301) UID: 0 2024-04-24T14:32:59.851 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:59.851 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:59.851 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:32:59.851 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:59.860 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:32:59.872 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:32:59.875+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:32:59.959 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:32:59.959 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:32:59.959 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:32:59.959 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:32:59.959 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:32:59.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:32:59.973 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:32:59.975+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:00.069 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:00.069 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:00.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:00.070 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:00.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.070 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.070 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:00.070 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.748126274999997, 2024-04-24T14:33:00.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011830553000000001, 2024-04-24T14:33:00.070 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.070 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.070 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.070 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.070 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.070 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 36 2024-04-24T14:33:00.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.071 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:00.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:00.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326000+0000", 2024-04-24T14:33:00.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:00.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326006+0000", 2024-04-24T14:33:00.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9340000000000003e-06 2024-04-24T14:33:00.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326044+0000", 2024-04-24T14:33:00.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8124999999999998e-05 2024-04-24T14:33:00.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326054+0000", 2024-04-24T14:33:00.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9650000000000003e-06 2024-04-24T14:33:00.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332746+0000", 2024-04-24T14:33:00.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006692539 2024-04-24T14:33:00.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332833+0000", 2024-04-24T14:33:00.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6482000000000003e-05 2024-04-24T14:33:00.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332947+0000", 2024-04-24T14:33:00.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114472 2024-04-24T14:33:00.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334320+0000", 2024-04-24T14:33:00.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001372489 2024-04-24T14:33:00.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337717+0000", 2024-04-24T14:33:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003396849 2024-04-24T14:33:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337789+0000", 2024-04-24T14:33:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2080000000000001e-05 2024-04-24T14:33:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337831+0000", 2024-04-24T14:33:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.244e-05 2024-04-24T14:33:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.075 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.076 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.076 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:00.076 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.748064413000002, 2024-04-24T14:33:00.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:33:00.076 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.076 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.076 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.076 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.076 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.076 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:33:00.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:33:00.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:00.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:33:00.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:33:00.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:33:00.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:33:00.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:33:00.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:33:00.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:33:00.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:33:00.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:33:00.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:33:00.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:33:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:33:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:33:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:33:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:33:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:33:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:33:00.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:33:00.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:33:00.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:33:00.081 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.081 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.081 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.082 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.082 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:00.082 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.654559939999999, 2024-04-24T14:33:00.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011218514000000001, 2024-04-24T14:33:00.082 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.082 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.082 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.082 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.082 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.082 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 88 2024-04-24T14:33:00.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.082 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:00.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:00.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419565+0000", 2024-04-24T14:33:00.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:00.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419575+0000", 2024-04-24T14:33:00.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5340000000000005e-06 2024-04-24T14:33:00.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419616+0000", 2024-04-24T14:33:00.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1301999999999999e-05 2024-04-24T14:33:00.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419638+0000", 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2177000000000001e-05 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428578+0000", 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089398280000000004 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428665+0000", 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6644000000000006e-05 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428797+0000", 2024-04-24T14:33:00.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013231500000000001 2024-04-24T14:33:00.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.429866+0000", 2024-04-24T14:33:00.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001069221 2024-04-24T14:33:00.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430684+0000", 2024-04-24T14:33:00.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081788399999999995 2024-04-24T14:33:00.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430743+0000", 2024-04-24T14:33:00.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8875999999999997e-05 2024-04-24T14:33:00.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430786+0000", 2024-04-24T14:33:00.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2398999999999998e-05 2024-04-24T14:33:00.087 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.087 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.087 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.088 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.088 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:00.088 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.448549471, 2024-04-24T14:33:00.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:33:00.088 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.088 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.088 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.088 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.088 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.088 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:33:00.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.088 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:00.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:00.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:33:00.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:00.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:33:00.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:33:00.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:33:00.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:33:00.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:33:00.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:33:00.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:33:00.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:33:00.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:33:00.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:33:00.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:33:00.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:33:00.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:33:00.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:33:00.093 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.093 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.093 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.093 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.093 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:00.093 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.448323703, 2024-04-24T14:33:00.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:33:00.094 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.094 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.094 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.094 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.094 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.094 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:33:00.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.094 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:00.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:00.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:33:00.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:00.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:33:00.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:33:00.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:33:00.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:33:00.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:33:00.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:33:00.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:33:00.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:33:00.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:33:00.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:33:00.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:33:00.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:33:00.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:33:00.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:33:00.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:33:00.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:33:00.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:33:00.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:33:00.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:33:00.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:33:00.099 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.099 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.099 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.099 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.099 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:00.100 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.448146616000002, 2024-04-24T14:33:00.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:33:00.100 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.100 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.100 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.100 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.100 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.100 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:33:00.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.100 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:00.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:00.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:33:00.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:00.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:33:00.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:33:00.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:33:00.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:33:00.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:33:00.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:33:00.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:33:00.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:33:00.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:33:00.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:33:00.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:33:00.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:33:00.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:33:00.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:33:00.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:33:00.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:33:00.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:33:00.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:33:00.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:33:00.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:33:00.105 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.105 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.105 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.105 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.105 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:00.106 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.448042702000002, 2024-04-24T14:33:00.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:33:00.106 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.106 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.106 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.106 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.106 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.106 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:33:00.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.106 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:00.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:00.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:33:00.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:00.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:33:00.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:33:00.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:33:00.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:33:00.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:33:00.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:33:00.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:33:00.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:33:00.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:33:00.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:33:00.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:33:00.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:33:00.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:33:00.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:33:00.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:33:00.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:33:00.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:33:00.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:33:00.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:33:00.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:33:00.111 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.111 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.111 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.111 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.111 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:00.111 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.447746381999998, 2024-04-24T14:33:00.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:33:00.112 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.112 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.112 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.112 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.112 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.112 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:33:00.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.112 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:00.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.113 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:00.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.113 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:33:00.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:00.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.113 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:33:00.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:33:00.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:33:00.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:33:00.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:33:00.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:33:00.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:33:00.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:33:00.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:33:00.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:33:00.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:33:00.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:33:00.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:33:00.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:33:00.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:33:00.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:33:00.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:33:00.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:33:00.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:33:00.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:33:00.117 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.117 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.117 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.117 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.118 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:00.118 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.447654251000003, 2024-04-24T14:33:00.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:33:00.118 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.118 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.118 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.118 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.118 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.118 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:33:00.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.118 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:00.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:00.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:33:00.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:00.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:33:00.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:33:00.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:33:00.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:33:00.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:33:00.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:33:00.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:33:00.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:33:00.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:33:00.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:33:00.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:33:00.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:33:00.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:33:00.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:33:00.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:33:00.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:33:00.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.447569039000001, 2024-04-24T14:33:00.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:33:00.124 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.124 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.124 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.124 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.124 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.124 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:33:00.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.124 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:00.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:33:00.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:33:00.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:33:00.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:33:00.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:33:00.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:33:00.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:33:00.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:33:00.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:33:00.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:33:00.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:33:00.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:33:00.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:33:00.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:33:00.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:33:00.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:33:00.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:33:00.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:33:00.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:33:00.129 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.129 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.129 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.129 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.129 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:00.129 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.447332158000002, 2024-04-24T14:33:00.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:33:00.129 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.130 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.130 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.130 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.130 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.130 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:33:00.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.130 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:00.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:00.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:33:00.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:00.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:33:00.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:33:00.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:33:00.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:33:00.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:33:00.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:33:00.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:33:00.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:33:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:33:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:33:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:33:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:33:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:33:00.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:33:00.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:33:00.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:33:00.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:33:00.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:33:00.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:33:00.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:33:00.135 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.135 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.135 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.135 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.135 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:00.135 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.447250636, 2024-04-24T14:33:00.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:33:00.135 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.135 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.136 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.136 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.136 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.136 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:33:00.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.136 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:00.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:33:00.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:33:00.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:33:00.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:33:00.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:33:00.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:33:00.140 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.015531559999999, 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01155111, 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135 2024-04-24T14:33:00.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.142 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:00.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:00.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058602+0000", 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1239999999999999e-06 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058609+0000", 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6719999999999998e-06 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058626+0000", 2024-04-24T14:33:00.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7569999999999999e-05 2024-04-24T14:33:00.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065271+0000", 2024-04-24T14:33:00.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00664469 2024-04-24T14:33:00.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065398+0000", 2024-04-24T14:33:00.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012694299999999999 2024-04-24T14:33:00.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065598+0000", 2024-04-24T14:33:00.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000200216 2024-04-24T14:33:00.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067596+0000", 2024-04-24T14:33:00.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019976450000000001 2024-04-24T14:33:00.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070055+0000", 2024-04-24T14:33:00.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024593710000000001 2024-04-24T14:33:00.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070109+0000", 2024-04-24T14:33:00.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3690999999999998e-05 2024-04-24T14:33:00.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070147+0000", 2024-04-24T14:33:00.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7910000000000001e-05 2024-04-24T14:33:00.146 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.146 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.146 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.147 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.147 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:00.147 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.015492225000003, 2024-04-24T14:33:00.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:33:00.147 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.147 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.147 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.147 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.147 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.147 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:33:00.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.147 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:33:00.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:33:00.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:33:00.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:33:00.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:33:00.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:33:00.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:33:00.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:33:00.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:33:00.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:33:00.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:33:00.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:33:00.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:33:00.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:33:00.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:33:00.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:33:00.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:00.152 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.015450721999997, 2024-04-24T14:33:00.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:33:00.153 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.153 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.153 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.153 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.153 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.153 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:33:00.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.153 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:00.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:00.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:00.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:00.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:33:00.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:33:00.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:33:00.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:33:00.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:33:00.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:33:00.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:33:00.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:33:00.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:33:00.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:33:00.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:33:00.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:33:00.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:33:00.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:33:00.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:33:00.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:33:00.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:33:00.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:33:00.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:33:00.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:33:00.158 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.158 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.158 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.158 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.158 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:00.158 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.015338288999999, 2024-04-24T14:33:00.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:00.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:33:00.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:00.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:33:00.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:33:00.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:33:00.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:33:00.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:33:00.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:33:00.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:33:00.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:33:00.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.015309989000002, 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:33:00.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.165 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:00.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:00.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:33:00.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:00.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:33:00.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:33:00.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:33:00.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:33:00.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:33:00.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:33:00.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:33:00.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:33:00.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:33:00.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:33:00.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:33:00.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:33:00.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:33:00.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:33:00.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.170 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.170 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:00.170 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.015222285, 2024-04-24T14:33:00.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:33:00.170 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.170 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.170 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.170 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.170 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.170 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:33:00.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.170 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:00.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:00.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:33:00.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:00.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:33:00.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:33:00.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:33:00.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:33:00.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:33:00.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:33:00.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.015099483999997, 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.175 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.176 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.176 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:33:00.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.176 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:00.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:00.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:33:00.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:33:00.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:33:00.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:33:00.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:33:00.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:33:00.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:33:00.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.371859551999997, 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:00.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:00.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:00.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:00.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:00.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:33:00.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:00.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:00.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:33:00.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:33:00.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:00.183 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:33:00.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:33:00.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.183 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:33:00.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:33:00.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.184 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:00.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:33:00.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:33:00.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:00.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:33:00.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:33:00.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:00.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:33:00.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:33:00.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:00.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:00.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:00.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:33:00.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:33:00.186 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.186 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.186 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.186 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:00.186 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:00.186 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:00.188 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:00.188 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:00.191+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:00.288 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:00.291+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:00.298 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-04-24T14:33:00.298 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:33:00.298 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3, 4, 5] 2024-04-24T14:33:00.298 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-24T14:33:00.298 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T14:33:00.339 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:00.339 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:00.339 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:00.339 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:00.349 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:00.352 INFO:tasks.ceph.osd.1:Stopped 2024-04-24T14:33:00.353 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-04-24T14:33:00.353 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi090.front.sepia.ceph.com 2024-04-24T14:33:00.353 DEBUG:teuthology.orchestra.run.smithi090:> type ceph-objectstore-tool 2024-04-24T14:33:00.356 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-24T14:33:00.356 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-04-24T14:33:00.389 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:00.391+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:00.446 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:00.447 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:00.447 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:00.447 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:00.447 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:00.463 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:00.491 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:00.495+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:00.561 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:00.561 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:00.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:00.561 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-24T14:33:00.561 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:00.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:00.592 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:00.595+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:00.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:33:00.603+0000 7f33c95bd180 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-04-24T14:33:00.667 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:00.667 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:00.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:00.723 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:00.727+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:00.794 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:00.795+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:00.832 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:00.832 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:00.833 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:00.833 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:00.833 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:00.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:00.895 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:00.899+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:00.937 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:00.937 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:00.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:00.937 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:00.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.937 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:36 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'22, mlcod=13'17)", 2024-04-24T14:33:00.937 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:00.938 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.609116012999998, 2024-04-24T14:33:00.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043791400000000001, 2024-04-24T14:33:00.938 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:00.938 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.938 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:00.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:00.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:00.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:00.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331851+0000", 2024-04-24T14:33:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331864+0000", 2024-04-24T14:33:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2767e-05 2024-04-24T14:33:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331881+0000", 2024-04-24T14:33:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6586e-05 2024-04-24T14:33:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331894+0000", 2024-04-24T14:33:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2976000000000001e-05 2024-04-24T14:33:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332161+0000", 2024-04-24T14:33:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026765399999999998 2024-04-24T14:33:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332197+0000", 2024-04-24T14:33:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5565999999999998e-05 2024-04-24T14:33:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336158+0000", 2024-04-24T14:33:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039616820000000002 2024-04-24T14:33:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336232+0000", 2024-04-24T14:33:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3151999999999994e-05 2024-04-24T14:33:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:37 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'23, mlcod=13'18)", 2024-04-24T14:33:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.608229535, 2024-04-24T14:33:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040417389999999999, 2024-04-24T14:33:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332738+0000", 2024-04-24T14:33:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332750+0000", 2024-04-24T14:33:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2432000000000001e-05 2024-04-24T14:33:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332768+0000", 2024-04-24T14:33:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7314000000000001e-05 2024-04-24T14:33:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332780+0000", 2024-04-24T14:33:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2716e-05 2024-04-24T14:33:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332925+0000", 2024-04-24T14:33:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144587 2024-04-24T14:33:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332955+0000", 2024-04-24T14:33:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0246000000000001e-05 2024-04-24T14:33:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336698+0000", 2024-04-24T14:33:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037430559999999998 2024-04-24T14:33:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336781+0000", 2024-04-24T14:33:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2447999999999994e-05 2024-04-24T14:33:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:39 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'25, mlcod=13'22)", 2024-04-24T14:33:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.602111211999997, 2024-04-24T14:33:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038736740000000001, 2024-04-24T14:33:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:00.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:00.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:00.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:00.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338856+0000", 2024-04-24T14:33:00.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:00.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:00.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338995+0000", 2024-04-24T14:33:00.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013868100000000001 2024-04-24T14:33:00.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:00.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339029+0000", 2024-04-24T14:33:00.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4372000000000002e-05 2024-04-24T14:33:00.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339050+0000", 2024-04-24T14:33:00.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0656999999999999e-05 2024-04-24T14:33:00.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:00.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339156+0000", 2024-04-24T14:33:00.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106177 2024-04-24T14:33:00.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:00.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339188+0000", 2024-04-24T14:33:00.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.243e-05 2024-04-24T14:33:00.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:00.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342648+0000", 2024-04-24T14:33:00.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034601689999999999 2024-04-24T14:33:00.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:00.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342731+0000", 2024-04-24T14:33:00.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2534999999999995e-05 2024-04-24T14:33:00.951 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:00.951 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:00.951 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:00.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.951 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:67 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'41, mlcod=13'39)", 2024-04-24T14:33:00.951 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:00.951 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.551374928000001, 2024-04-24T14:33:00.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031029009999999999, 2024-04-24T14:33:00.951 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:00.951 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.951 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389592+0000", 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:00.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:00.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389603+0000", 2024-04-24T14:33:00.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090000000000001e-05 2024-04-24T14:33:00.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:00.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389631+0000", 2024-04-24T14:33:00.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7772000000000001e-05 2024-04-24T14:33:00.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:00.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389646+0000", 2024-04-24T14:33:00.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.432e-05 2024-04-24T14:33:00.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:00.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389732+0000", 2024-04-24T14:33:00.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6331999999999999e-05 2024-04-24T14:33:00.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:00.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389771+0000", 2024-04-24T14:33:00.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8751999999999998e-05 2024-04-24T14:33:00.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:00.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392585+0000", 2024-04-24T14:33:00.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028147469999999998 2024-04-24T14:33:00.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:00.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392696+0000", 2024-04-24T14:33:00.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111121 2024-04-24T14:33:00.956 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:00.956 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:00.956 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:00.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.956 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T14:33:00.956 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:00.956 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.550232317000003, 2024-04-24T14:33:00.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030475519999999998, 2024-04-24T14:33:00.956 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:00.956 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:00.956 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:00.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:00.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:00.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:00.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:00.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:00.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:00.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:00.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390735+0000", 2024-04-24T14:33:00.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:00.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:00.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390804+0000", 2024-04-24T14:33:00.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9488999999999998e-05 2024-04-24T14:33:00.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:00.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:00.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390832+0000", 2024-04-24T14:33:00.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7946e-05 2024-04-24T14:33:00.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:00.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390847+0000", 2024-04-24T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4693e-05 2024-04-24T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391508+0000", 2024-04-24T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066154799999999998 2024-04-24T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391538+0000", 2024-04-24T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0029e-05 2024-04-24T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393698+0000", 2024-04-24T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002160052 2024-04-24T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393784+0000", 2024-04-24T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5414999999999995e-05 2024-04-24T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:72 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'46, mlcod=13'43)", 2024-04-24T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.544058823999997, 2024-04-24T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036059460000000001, 2024-04-24T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396908+0000", 2024-04-24T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396968+0000", 2024-04-24T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9412e-05 2024-04-24T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397010+0000", 2024-04-24T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2169000000000002e-05 2024-04-24T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397022+0000", 2024-04-24T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1749999999999999e-05 2024-04-24T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397096+0000", 2024-04-24T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4701000000000002e-05 2024-04-24T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397126+0000", 2024-04-24T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2024-04-24T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400443+0000", 2024-04-24T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033172140000000002 2024-04-24T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400516+0000", 2024-04-24T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2181999999999994e-05 2024-04-24T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:73 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'47, mlcod=13'43)", 2024-04-24T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.543211419999999, 2024-04-24T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055152170000000002, 2024-04-24T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397756+0000", 2024-04-24T14:33:01.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:01.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397884+0000", 2024-04-24T14:33:01.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012842100000000001 2024-04-24T14:33:01.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397923+0000", 2024-04-24T14:33:01.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8942000000000001e-05 2024-04-24T14:33:01.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397939+0000", 2024-04-24T14:33:01.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6073e-05 2024-04-24T14:33:01.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398106+0000", 2024-04-24T14:33:01.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016737100000000001 2024-04-24T14:33:01.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398137+0000", 2024-04-24T14:33:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0496000000000001e-05 2024-04-24T14:33:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403191+0000", 2024-04-24T14:33:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00505391 2024-04-24T14:33:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403272+0000", 2024-04-24T14:33:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1483999999999994e-05 2024-04-24T14:33:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:74 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'48, mlcod=13'43)", 2024-04-24T14:33:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.542412394000003, 2024-04-24T14:33:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048091760000000001, 2024-04-24T14:33:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398555+0000", 2024-04-24T14:33:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398570+0000", 2024-04-24T14:33:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.558e-05 2024-04-24T14:33:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398610+0000", 2024-04-24T14:33:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0028999999999999e-05 2024-04-24T14:33:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398625+0000", 2024-04-24T14:33:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-04-24T14:33:01.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398701+0000", 2024-04-24T14:33:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6507000000000005e-05 2024-04-24T14:33:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398731+0000", 2024-04-24T14:33:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9356000000000002e-05 2024-04-24T14:33:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403281+0000", 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004549796 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403365+0000", 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4685000000000007e-05 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.521099468999999, 2024-04-24T14:33:01.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038140460000000002, 2024-04-24T14:33:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419868+0000", 2024-04-24T14:33:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419882+0000", 2024-04-24T14:33:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.45e-05 2024-04-24T14:33:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419919+0000", 2024-04-24T14:33:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6939e-05 2024-04-24T14:33:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419961+0000", 2024-04-24T14:33:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2148999999999999e-05 2024-04-24T14:33:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420084+0000", 2024-04-24T14:33:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122958 2024-04-24T14:33:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420122+0000", 2024-04-24T14:33:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8334999999999998e-05 2024-04-24T14:33:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423616+0000", 2024-04-24T14:33:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003493566 2024-04-24T14:33:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423683+0000", 2024-04-24T14:33:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7149000000000004e-05 2024-04-24T14:33:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T14:33:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.519687714, 2024-04-24T14:33:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030290730000000002, 2024-04-24T14:33:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421279+0000", 2024-04-24T14:33:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421352+0000", 2024-04-24T14:33:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2251999999999999e-05 2024-04-24T14:33:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421378+0000", 2024-04-24T14:33:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6006000000000001e-05 2024-04-24T14:33:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421388+0000", 2024-04-24T14:33:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0373e-05 2024-04-24T14:33:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421473+0000", 2024-04-24T14:33:01.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4665000000000003e-05 2024-04-24T14:33:01.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421512+0000", 2024-04-24T14:33:01.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9619000000000001e-05 2024-04-24T14:33:01.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424198+0000", 2024-04-24T14:33:01.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026854610000000001 2024-04-24T14:33:01.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424310+0000", 2024-04-24T14:33:01.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011208 2024-04-24T14:33:01.050 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.050 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.050 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.050 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:87 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'57, mlcod=13'51)", 2024-04-24T14:33:01.050 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:01.050 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.515424754999998, 2024-04-24T14:33:01.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002974057, 2024-04-24T14:33:01.050 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.050 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425542+0000", 2024-04-24T14:33:01.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:01.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425614+0000", 2024-04-24T14:33:01.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1328999999999994e-05 2024-04-24T14:33:01.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425664+0000", 2024-04-24T14:33:01.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0491999999999997e-05 2024-04-24T14:33:01.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425676+0000", 2024-04-24T14:33:01.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.134e-05 2024-04-24T14:33:01.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425798+0000", 2024-04-24T14:33:01.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012221700000000001 2024-04-24T14:33:01.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425860+0000", 2024-04-24T14:33:01.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1761999999999996e-05 2024-04-24T14:33:01.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428368+0000", 2024-04-24T14:33:01.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025084 2024-04-24T14:33:01.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428518+0000", 2024-04-24T14:33:01.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149884 2024-04-24T14:33:01.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.057 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-04-24T14:33:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.315316029999998, 2024-04-24T14:33:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002919694, 2024-04-24T14:33:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625651+0000", 2024-04-24T14:33:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625667+0000", 2024-04-24T14:33:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5973000000000001e-05 2024-04-24T14:33:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625689+0000", 2024-04-24T14:33:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:33:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625705+0000", 2024-04-24T14:33:01.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5797000000000001e-05 2024-04-24T14:33:01.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625805+0000", 2024-04-24T14:33:01.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9238000000000001e-05 2024-04-24T14:33:01.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625837+0000", 2024-04-24T14:33:01.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2722000000000003e-05 2024-04-24T14:33:01.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628476+0000", 2024-04-24T14:33:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026391980000000002 2024-04-24T14:33:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628572+0000", 2024-04-24T14:33:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5754000000000005e-05 2024-04-24T14:33:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'88)", 2024-04-24T14:33:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.875378152000003, 2024-04-24T14:33:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031994179999999999, 2024-04-24T14:33:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065589+0000", 2024-04-24T14:33:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065602+0000", 2024-04-24T14:33:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2737e-05 2024-04-24T14:33:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065620+0000", 2024-04-24T14:33:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8546000000000002e-05 2024-04-24T14:33:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065629+0000", 2024-04-24T14:33:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1600000000000004e-06 2024-04-24T14:33:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065729+0000", 2024-04-24T14:33:01.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9933999999999996e-05 2024-04-24T14:33:01.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065792+0000", 2024-04-24T14:33:01.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2712000000000005e-05 2024-04-24T14:33:01.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068667+0000", 2024-04-24T14:33:01.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028752920000000002 2024-04-24T14:33:01.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068790+0000", 2024-04-24T14:33:01.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012244400000000001 2024-04-24T14:33:01.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.067 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.067 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:137 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'91)", 2024-04-24T14:33:01.067 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:01.067 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.874022416000003, 2024-04-24T14:33:01.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052524399999999997, 2024-04-24T14:33:01.067 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.068 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.068 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:01.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:01.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066945+0000", 2024-04-24T14:33:01.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:01.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067031+0000", 2024-04-24T14:33:01.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6228000000000002e-05 2024-04-24T14:33:01.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067060+0000", 2024-04-24T14:33:01.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8445999999999998e-05 2024-04-24T14:33:01.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067069+0000", 2024-04-24T14:33:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999996e-06 2024-04-24T14:33:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067172+0000", 2024-04-24T14:33:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010285099999999999 2024-04-24T14:33:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067234+0000", 2024-04-24T14:33:01.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2068000000000003e-05 2024-04-24T14:33:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072090+0000", 2024-04-24T14:33:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048560859999999999 2024-04-24T14:33:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072199+0000", 2024-04-24T14:33:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108384 2024-04-24T14:33:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.072 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:138 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'96, mlcod=13'91)", 2024-04-24T14:33:01.072 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:01.072 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.873022652000003, 2024-04-24T14:33:01.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053159380000000001, 2024-04-24T14:33:01.072 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.072 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.072 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:01.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:01.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067945+0000", 2024-04-24T14:33:01.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:01.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067957+0000", 2024-04-24T14:33:01.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1963e-05 2024-04-24T14:33:01.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067982+0000", 2024-04-24T14:33:01.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5570000000000001e-05 2024-04-24T14:33:01.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067994+0000", 2024-04-24T14:33:01.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2306e-05 2024-04-24T14:33:01.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068282+0000", 2024-04-24T14:33:01.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028799000000000001 2024-04-24T14:33:01.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068307+0000", 2024-04-24T14:33:01.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4351000000000001e-05 2024-04-24T14:33:01.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073142+0000", 2024-04-24T14:33:01.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004835326 2024-04-24T14:33:01.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073262+0000", 2024-04-24T14:33:01.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119695 2024-04-24T14:33:01.076 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.076 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.076 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.076 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:139 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T14:33:01.076 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:01.076 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.871539824999999, 2024-04-24T14:33:01.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039447479999999997, 2024-04-24T14:33:01.076 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.076 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069428+0000", 2024-04-24T14:33:01.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:01.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069440+0000", 2024-04-24T14:33:01.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2047e-05 2024-04-24T14:33:01.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069469+0000", 2024-04-24T14:33:01.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8852000000000001e-05 2024-04-24T14:33:01.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069477+0000", 2024-04-24T14:33:01.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-04-24T14:33:01.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069531+0000", 2024-04-24T14:33:01.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4126999999999999e-05 2024-04-24T14:33:01.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069559+0000", 2024-04-24T14:33:01.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7881e-05 2024-04-24T14:33:01.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073270+0000", 2024-04-24T14:33:01.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037104519999999999 2024-04-24T14:33:01.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073373+0000", 2024-04-24T14:33:01.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010380899999999999 2024-04-24T14:33:01.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.080 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:140 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'92)", 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.870713948999999, 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031722510000000001, 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:01.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070253+0000", 2024-04-24T14:33:01.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:01.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070307+0000", 2024-04-24T14:33:01.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3909000000000002e-05 2024-04-24T14:33:01.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070330+0000", 2024-04-24T14:33:01.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2599000000000002e-05 2024-04-24T14:33:01.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070337+0000", 2024-04-24T14:33:01.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3030000000000002e-06 2024-04-24T14:33:01.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070377+0000", 2024-04-24T14:33:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9916000000000002e-05 2024-04-24T14:33:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070398+0000", 2024-04-24T14:33:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0579000000000001e-05 2024-04-24T14:33:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073377+0000", 2024-04-24T14:33:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002979013 2024-04-24T14:33:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073427+0000", 2024-04-24T14:33:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0078999999999999e-05 2024-04-24T14:33:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:143 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'101, mlcod=13'98)", 2024-04-24T14:33:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.864528669000002, 2024-04-24T14:33:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030247590000000001, 2024-04-24T14:33:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076438+0000", 2024-04-24T14:33:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076533+0000", 2024-04-24T14:33:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4443000000000004e-05 2024-04-24T14:33:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076563+0000", 2024-04-24T14:33:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0386999999999999e-05 2024-04-24T14:33:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076604+0000", 2024-04-24T14:33:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1012e-05 2024-04-24T14:33:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076711+0000", 2024-04-24T14:33:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010637300000000001 2024-04-24T14:33:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076737+0000", 2024-04-24T14:33:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6190000000000002e-05 2024-04-24T14:33:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079368+0000", 2024-04-24T14:33:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026313529999999999 2024-04-24T14:33:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079465+0000", 2024-04-24T14:33:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6426000000000001e-05 2024-04-24T14:33:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:159 1.0 e14/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'113, mlcod=14'109)", 2024-04-24T14:33:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.230708563, 2024-04-24T14:33:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052053589999999997, 2024-04-24T14:33:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710259+0000", 2024-04-24T14:33:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710272+0000", 2024-04-24T14:33:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.382e-05 2024-04-24T14:33:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710303+0000", 2024-04-24T14:33:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0629000000000001e-05 2024-04-24T14:33:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710322+0000", 2024-04-24T14:33:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8859999999999999e-05 2024-04-24T14:33:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710479+0000", 2024-04-24T14:33:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157076 2024-04-24T14:33:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710535+0000", 2024-04-24T14:33:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6289000000000003e-05 2024-04-24T14:33:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715391+0000", 2024-04-24T14:33:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048557139999999997 2024-04-24T14:33:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715465+0000", 2024-04-24T14:33:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4301000000000006e-05 2024-04-24T14:33:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:161 1.0 e14/12 1:30306672:devicehealth::main.db.0000000000000000:head v 14'115, mlcod=14'112)", 2024-04-24T14:33:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.223597761000001, 2024-04-24T14:33:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039960450000000002, 2024-04-24T14:33:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717370+0000", 2024-04-24T14:33:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717499+0000", 2024-04-24T14:33:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012877400000000001 2024-04-24T14:33:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717513+0000", 2024-04-24T14:33:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4229999999999999e-05 2024-04-24T14:33:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717521+0000", 2024-04-24T14:33:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2024-04-24T14:33:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718040+0000", 2024-04-24T14:33:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051920099999999995 2024-04-24T14:33:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718094+0000", 2024-04-24T14:33:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859000000000001e-05 2024-04-24T14:33:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721278+0000", 2024-04-24T14:33:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031838560000000001 2024-04-24T14:33:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721367+0000", 2024-04-24T14:33:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9241999999999997e-05 2024-04-24T14:33:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:01.098 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:01.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:01.100 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:01.103+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:01.201 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:01.203+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:01.249 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:01.249 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:01.249 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:01.249 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:01.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:01.301 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:01.303+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:01.363 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:01.364 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:01.364 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:01.364 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:01.364 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:01.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:01.401 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:01.403+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:01.470 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:01.470 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:01.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:01.470 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-04-24T14:33:01.470 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:01.479 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:01.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T14:33:01.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:33:01.527 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:01.531+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:01.583 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:01.583 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:01.583 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:01.583 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:01.592 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:01.604 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:01.607+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:01.642 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:01.661 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:01.682 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:01.688 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:01.688 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:01.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-04-24T14:33:01.691+0000 7f33c95bd180 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-04-24T14:33:01.698 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:01.704 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:01.724 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:01.724 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:01.724 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:01.724 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:01.724 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:01.724 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:01.724 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:01.727 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:01.743 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:01.747+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:01.797 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:01.797 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:01.806 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:01.807+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:01.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:01.903 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:01.904 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:01.906 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:01.907+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:01.912 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:2.6 2024-04-24T14:33:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:2.4 2024-04-24T14:33:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:3.0 2024-04-24T14:33:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:2.1 2024-04-24T14:33:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:3.2 2024-04-24T14:33:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:2.2 2024-04-24T14:33:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:2.0 2024-04-24T14:33:02.006 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:02.011+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:02.011 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:02.012 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:02.021 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:02.108 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:02.111+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:02.164 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:02.164 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:02.173 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:02.223 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:02.227+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:02.275 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:02.276 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:02.286 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:02.309 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:02.311+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:02.387 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:02.388 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:02.396 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:02.410 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:02.411+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:02.495 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:02.495 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:02.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:02.511 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-24T14:33:02.547 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:02.551+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:02.553 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 3.2 --file /var/log/ceph/exp.3.2.1 2024-04-24T14:33:02.605 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:02.605 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:02.611 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:02.615+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:02.614 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:02.708 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:02.709 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:02.712 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:02.715+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:02.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:02.813 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:02.815+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:02.816 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:02.816 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:02.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:02.914 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:02.915+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:02.973 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:02.973 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:02.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:03.014 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:03.019+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:03.084 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:03.084 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:03.093 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:03.115 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:03.119+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:03.186 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:03.186 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:03.195 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:03.239 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:03.243+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:03.307 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:03.307 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:03.317 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:03.317 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:03.319+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:03.417 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:03.419+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:03.421 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:03.421 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:03.430 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:03.518 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:03.519+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:03.535 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:03.535 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:03.543 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:03.619 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:03.623+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:03.697 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:03.697 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:03.706 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:03.719 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:03.723+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:03.721 INFO:teuthology.orchestra.run.smithi090.stderr:Exporting 3.2 info 3.2( empty local-lis/les=18/19 n=0 ec=18/18 lis/c=18/18 les/c/f=19/19/0 sis=18) 2024-04-24T14:33:03.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:33:03.728 INFO:teuthology.orchestra.run.smithi090.stderr:Export successful 2024-04-24T14:33:03.802 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:03.802 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:03.811 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:03.819 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:03.823+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:03.856 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:03.875 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:03.896 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:03.913 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:03.913 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:03.918 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:03.920 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:03.923+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:03.922 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:03.941 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:03.942 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:03.942 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:03.942 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:03.942 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:03.942 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:03.942 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:03.946 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:03.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:04.021 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:04.023+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:04.068 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:04.068 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:04.068 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:04.077 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:04.123 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:04.127+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:04.172 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:04.172 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:04.172 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:04.172 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:04.172 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:04.181 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:04.223 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:04.227+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:04.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 3.2 2024-04-24T14:33:04.277 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:04.277 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:04.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:04.277 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-24T14:33:04.277 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:04.286 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:04.327 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:04.331+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:04.389 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:04.389 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:04.389 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:04.389 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:04.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:04.443 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:04.447+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:04.493 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:04.493 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:04.493 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:04.493 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:04.493 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:04.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:04.524 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T14:33:04.575 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:04.579+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:04.605 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.605 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:36 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'22, mlcod=13'17)", 2024-04-24T14:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.274871777000001, 2024-04-24T14:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043791400000000001, 2024-04-24T14:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331851+0000", 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331864+0000", 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2767e-05 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331881+0000", 2024-04-24T14:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6586e-05 2024-04-24T14:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331894+0000", 2024-04-24T14:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2976000000000001e-05 2024-04-24T14:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332161+0000", 2024-04-24T14:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026765399999999998 2024-04-24T14:33:04.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332197+0000", 2024-04-24T14:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5565999999999998e-05 2024-04-24T14:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336158+0000", 2024-04-24T14:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039616820000000002 2024-04-24T14:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336232+0000", 2024-04-24T14:33:04.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3151999999999994e-05 2024-04-24T14:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:37 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'23, mlcod=13'18)", 2024-04-24T14:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.273985299000003, 2024-04-24T14:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040417389999999999, 2024-04-24T14:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332738+0000", 2024-04-24T14:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:04.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332750+0000", 2024-04-24T14:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2432000000000001e-05 2024-04-24T14:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332768+0000", 2024-04-24T14:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7314000000000001e-05 2024-04-24T14:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332780+0000", 2024-04-24T14:33:04.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2716e-05 2024-04-24T14:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332925+0000", 2024-04-24T14:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144587 2024-04-24T14:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332955+0000", 2024-04-24T14:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0246000000000001e-05 2024-04-24T14:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336698+0000", 2024-04-24T14:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037430559999999998 2024-04-24T14:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336781+0000", 2024-04-24T14:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2447999999999994e-05 2024-04-24T14:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:39 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'25, mlcod=13'22)", 2024-04-24T14:33:04.614 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.267866976000001, 2024-04-24T14:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038736740000000001, 2024-04-24T14:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:04.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338856+0000", 2024-04-24T14:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338995+0000", 2024-04-24T14:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013868100000000001 2024-04-24T14:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339029+0000", 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4372000000000002e-05 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339050+0000", 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0656999999999999e-05 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339156+0000", 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106177 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339188+0000", 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.243e-05 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342648+0000", 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034601689999999999 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342731+0000", 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2534999999999995e-05 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:67 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'41, mlcod=13'39)", 2024-04-24T14:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.217130691999998, 2024-04-24T14:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031029009999999999, 2024-04-24T14:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389592+0000", 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389603+0000", 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090000000000001e-05 2024-04-24T14:33:04.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389631+0000", 2024-04-24T14:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7772000000000001e-05 2024-04-24T14:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389646+0000", 2024-04-24T14:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.432e-05 2024-04-24T14:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389732+0000", 2024-04-24T14:33:04.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6331999999999999e-05 2024-04-24T14:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389771+0000", 2024-04-24T14:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8751999999999998e-05 2024-04-24T14:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392585+0000", 2024-04-24T14:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028147469999999998 2024-04-24T14:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392696+0000", 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111121 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.215988080999999, 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030475519999999998, 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390735+0000", 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390804+0000", 2024-04-24T14:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9488999999999998e-05 2024-04-24T14:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390832+0000", 2024-04-24T14:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7946e-05 2024-04-24T14:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390847+0000", 2024-04-24T14:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4693e-05 2024-04-24T14:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391508+0000", 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066154799999999998 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391538+0000", 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0029e-05 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393698+0000", 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002160052 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393784+0000", 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5414999999999995e-05 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:72 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'46, mlcod=13'43)", 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.209814588, 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036059460000000001, 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.627 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396908+0000", 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396968+0000", 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9412e-05 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397010+0000", 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2169000000000002e-05 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397022+0000", 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1749999999999999e-05 2024-04-24T14:33:04.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397096+0000", 2024-04-24T14:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4701000000000002e-05 2024-04-24T14:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397126+0000", 2024-04-24T14:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2024-04-24T14:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400443+0000", 2024-04-24T14:33:04.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033172140000000002 2024-04-24T14:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400516+0000", 2024-04-24T14:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2181999999999994e-05 2024-04-24T14:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:73 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'47, mlcod=13'43)", 2024-04-24T14:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.208967184000002, 2024-04-24T14:33:04.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055152170000000002, 2024-04-24T14:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397756+0000", 2024-04-24T14:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397884+0000", 2024-04-24T14:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012842100000000001 2024-04-24T14:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397923+0000", 2024-04-24T14:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8942000000000001e-05 2024-04-24T14:33:04.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397939+0000", 2024-04-24T14:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6073e-05 2024-04-24T14:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398106+0000", 2024-04-24T14:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016737100000000001 2024-04-24T14:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398137+0000", 2024-04-24T14:33:04.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0496000000000001e-05 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403191+0000", 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00505391 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403272+0000", 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1483999999999994e-05 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:74 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'48, mlcod=13'43)", 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.208168157999999, 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048091760000000001, 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398555+0000", 2024-04-24T14:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398570+0000", 2024-04-24T14:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.558e-05 2024-04-24T14:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398610+0000", 2024-04-24T14:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0028999999999999e-05 2024-04-24T14:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398625+0000", 2024-04-24T14:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-04-24T14:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398701+0000", 2024-04-24T14:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6507000000000005e-05 2024-04-24T14:33:04.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398731+0000", 2024-04-24T14:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9356000000000002e-05 2024-04-24T14:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403281+0000", 2024-04-24T14:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004549796 2024-04-24T14:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403365+0000", 2024-04-24T14:33:04.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4685000000000007e-05 2024-04-24T14:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-04-24T14:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.186855233000003, 2024-04-24T14:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038140460000000002, 2024-04-24T14:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419868+0000", 2024-04-24T14:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419882+0000", 2024-04-24T14:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.45e-05 2024-04-24T14:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419919+0000", 2024-04-24T14:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6939e-05 2024-04-24T14:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419961+0000", 2024-04-24T14:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2148999999999999e-05 2024-04-24T14:33:04.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420084+0000", 2024-04-24T14:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122958 2024-04-24T14:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420122+0000", 2024-04-24T14:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8334999999999998e-05 2024-04-24T14:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423616+0000", 2024-04-24T14:33:04.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003493566 2024-04-24T14:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423683+0000", 2024-04-24T14:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7149000000000004e-05 2024-04-24T14:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T14:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.185443478000003, 2024-04-24T14:33:04.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030290730000000002, 2024-04-24T14:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421279+0000", 2024-04-24T14:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421352+0000", 2024-04-24T14:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2251999999999999e-05 2024-04-24T14:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421378+0000", 2024-04-24T14:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6006000000000001e-05 2024-04-24T14:33:04.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421388+0000", 2024-04-24T14:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0373e-05 2024-04-24T14:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421473+0000", 2024-04-24T14:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4665000000000003e-05 2024-04-24T14:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421512+0000", 2024-04-24T14:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9619000000000001e-05 2024-04-24T14:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424198+0000", 2024-04-24T14:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026854610000000001 2024-04-24T14:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424310+0000", 2024-04-24T14:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011208 2024-04-24T14:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.648 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:87 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'57, mlcod=13'51)", 2024-04-24T14:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.181180519000002, 2024-04-24T14:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002974057, 2024-04-24T14:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:04.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425542+0000", 2024-04-24T14:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425614+0000", 2024-04-24T14:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1328999999999994e-05 2024-04-24T14:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425664+0000", 2024-04-24T14:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0491999999999997e-05 2024-04-24T14:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425676+0000", 2024-04-24T14:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.134e-05 2024-04-24T14:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425798+0000", 2024-04-24T14:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012221700000000001 2024-04-24T14:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425860+0000", 2024-04-24T14:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1761999999999996e-05 2024-04-24T14:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428368+0000", 2024-04-24T14:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025084 2024-04-24T14:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428518+0000", 2024-04-24T14:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149884 2024-04-24T14:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-04-24T14:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.981071794000002, 2024-04-24T14:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002919694, 2024-04-24T14:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.653 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625651+0000", 2024-04-24T14:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625667+0000", 2024-04-24T14:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5973000000000001e-05 2024-04-24T14:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625689+0000", 2024-04-24T14:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625705+0000", 2024-04-24T14:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5797000000000001e-05 2024-04-24T14:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625805+0000", 2024-04-24T14:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9238000000000001e-05 2024-04-24T14:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625837+0000", 2024-04-24T14:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2722000000000003e-05 2024-04-24T14:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628476+0000", 2024-04-24T14:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026391980000000002 2024-04-24T14:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628572+0000", 2024-04-24T14:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5754000000000005e-05 2024-04-24T14:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.657 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'88)", 2024-04-24T14:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.541133916, 2024-04-24T14:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031994179999999999, 2024-04-24T14:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065589+0000", 2024-04-24T14:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065602+0000", 2024-04-24T14:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2737e-05 2024-04-24T14:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065620+0000", 2024-04-24T14:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8546000000000002e-05 2024-04-24T14:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065629+0000", 2024-04-24T14:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1600000000000004e-06 2024-04-24T14:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065729+0000", 2024-04-24T14:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9933999999999996e-05 2024-04-24T14:33:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065792+0000", 2024-04-24T14:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2712000000000005e-05 2024-04-24T14:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068667+0000", 2024-04-24T14:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028752920000000002 2024-04-24T14:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068790+0000", 2024-04-24T14:33:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012244400000000001 2024-04-24T14:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:137 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'91)", 2024-04-24T14:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.539778179999999, 2024-04-24T14:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052524399999999997, 2024-04-24T14:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066945+0000", 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067031+0000", 2024-04-24T14:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6228000000000002e-05 2024-04-24T14:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067060+0000", 2024-04-24T14:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8445999999999998e-05 2024-04-24T14:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067069+0000", 2024-04-24T14:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999996e-06 2024-04-24T14:33:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067172+0000", 2024-04-24T14:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010285099999999999 2024-04-24T14:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067234+0000", 2024-04-24T14:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2068000000000003e-05 2024-04-24T14:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072090+0000", 2024-04-24T14:33:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048560859999999999 2024-04-24T14:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072199+0000", 2024-04-24T14:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108384 2024-04-24T14:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:138 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'96, mlcod=13'91)", 2024-04-24T14:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.538778416, 2024-04-24T14:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053159380000000001, 2024-04-24T14:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067945+0000", 2024-04-24T14:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067957+0000", 2024-04-24T14:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1963e-05 2024-04-24T14:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067982+0000", 2024-04-24T14:33:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5570000000000001e-05 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067994+0000", 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2306e-05 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068282+0000", 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028799000000000001 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068307+0000", 2024-04-24T14:33:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4351000000000001e-05 2024-04-24T14:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073142+0000", 2024-04-24T14:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004835326 2024-04-24T14:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073262+0000", 2024-04-24T14:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119695 2024-04-24T14:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:139 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T14:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.537295589000003, 2024-04-24T14:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039447479999999997, 2024-04-24T14:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069428+0000", 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069440+0000", 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2047e-05 2024-04-24T14:33:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069469+0000", 2024-04-24T14:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8852000000000001e-05 2024-04-24T14:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069477+0000", 2024-04-24T14:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-04-24T14:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069531+0000", 2024-04-24T14:33:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4126999999999999e-05 2024-04-24T14:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069559+0000", 2024-04-24T14:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7881e-05 2024-04-24T14:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073270+0000", 2024-04-24T14:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037104519999999999 2024-04-24T14:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073373+0000", 2024-04-24T14:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010380899999999999 2024-04-24T14:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:140 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'92)", 2024-04-24T14:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.536469713000002, 2024-04-24T14:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031722510000000001, 2024-04-24T14:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070253+0000", 2024-04-24T14:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070307+0000", 2024-04-24T14:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3909000000000002e-05 2024-04-24T14:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070330+0000", 2024-04-24T14:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2599000000000002e-05 2024-04-24T14:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070337+0000", 2024-04-24T14:33:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3030000000000002e-06 2024-04-24T14:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070377+0000", 2024-04-24T14:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9916000000000002e-05 2024-04-24T14:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070398+0000", 2024-04-24T14:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0579000000000001e-05 2024-04-24T14:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073377+0000", 2024-04-24T14:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002979013 2024-04-24T14:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073427+0000", 2024-04-24T14:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0078999999999999e-05 2024-04-24T14:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:143 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'101, mlcod=13'98)", 2024-04-24T14:33:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.530284432999999, 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030247590000000001, 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076438+0000", 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076533+0000", 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4443000000000004e-05 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076563+0000", 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0386999999999999e-05 2024-04-24T14:33:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076604+0000", 2024-04-24T14:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1012e-05 2024-04-24T14:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076711+0000", 2024-04-24T14:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010637300000000001 2024-04-24T14:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076737+0000", 2024-04-24T14:33:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6190000000000002e-05 2024-04-24T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079368+0000", 2024-04-24T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026313529999999999 2024-04-24T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079465+0000", 2024-04-24T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6426000000000001e-05 2024-04-24T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:159 1.0 e14/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'113, mlcod=14'109)", 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.896464326999997, 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052053589999999997, 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710259+0000", 2024-04-24T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710272+0000", 2024-04-24T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.382e-05 2024-04-24T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710303+0000", 2024-04-24T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0629000000000001e-05 2024-04-24T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710322+0000", 2024-04-24T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8859999999999999e-05 2024-04-24T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710479+0000", 2024-04-24T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157076 2024-04-24T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710535+0000", 2024-04-24T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6289000000000003e-05 2024-04-24T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715391+0000", 2024-04-24T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048557139999999997 2024-04-24T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715465+0000", 2024-04-24T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4301000000000006e-05 2024-04-24T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:161 1.0 e14/12 1:30306672:devicehealth::main.db.0000000000000000:head v 14'115, mlcod=14'112)", 2024-04-24T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.889353524999997, 2024-04-24T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039960450000000002, 2024-04-24T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717370+0000", 2024-04-24T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717499+0000", 2024-04-24T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012877400000000001 2024-04-24T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717513+0000", 2024-04-24T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4229999999999999e-05 2024-04-24T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717521+0000", 2024-04-24T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2024-04-24T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718040+0000", 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051920099999999995 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718094+0000", 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859000000000001e-05 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721278+0000", 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031838560000000001 2024-04-24T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721367+0000", 2024-04-24T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9241999999999997e-05 2024-04-24T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:04.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:04.694 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:04.699+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:04.789 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:04.790 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:04.790 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:04.790 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:04.794 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:04.799+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:04.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:04.893 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:04.893 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:04.893 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:04.894 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:04.894 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:04.895 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:04.899+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:04.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:04.996 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:04.999+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:04.997 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:04.997 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:04.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:04.997 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-04-24T14:33:04.997 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:05.006 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:05.096 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:05.099+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:05.113 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:05.113 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:05.113 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:05.113 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:05.123 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:05.198 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:05.199+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:05.217 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:05.217 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:05.218 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:05.218 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:05.218 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:05.227 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:05.299 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:05.303+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:05.329 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:05.329 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:05.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:05.330 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:05.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.330 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.330 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:05.330 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 43.006351066000001, 2024-04-24T14:33:05.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011830553000000001, 2024-04-24T14:33:05.330 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.330 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.330 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.330 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.330 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 36 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326000+0000", 2024-04-24T14:33:05.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:05.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326006+0000", 2024-04-24T14:33:05.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9340000000000003e-06 2024-04-24T14:33:05.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326044+0000", 2024-04-24T14:33:05.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8124999999999998e-05 2024-04-24T14:33:05.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326054+0000", 2024-04-24T14:33:05.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9650000000000003e-06 2024-04-24T14:33:05.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332746+0000", 2024-04-24T14:33:05.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006692539 2024-04-24T14:33:05.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332833+0000", 2024-04-24T14:33:05.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6482000000000003e-05 2024-04-24T14:33:05.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332947+0000", 2024-04-24T14:33:05.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114472 2024-04-24T14:33:05.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334320+0000", 2024-04-24T14:33:05.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001372489 2024-04-24T14:33:05.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337717+0000", 2024-04-24T14:33:05.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003396849 2024-04-24T14:33:05.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337789+0000", 2024-04-24T14:33:05.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2080000000000001e-05 2024-04-24T14:33:05.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337831+0000", 2024-04-24T14:33:05.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.244e-05 2024-04-24T14:33:05.335 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.335 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.335 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.336 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.336 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:05.336 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 43.006289203999998, 2024-04-24T14:33:05.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:33:05.336 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.336 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.336 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.336 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.336 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.336 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:33:05.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.336 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:05.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:33:05.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:33:05.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:33:05.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:33:05.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:33:05.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:33:05.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:33:05.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:33:05.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:33:05.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:33:05.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.341 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:05.342 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.912784731000002, 2024-04-24T14:33:05.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011218514000000001, 2024-04-24T14:33:05.342 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.342 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.342 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.342 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.342 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.342 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 88 2024-04-24T14:33:05.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.342 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:05.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:05.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419565+0000", 2024-04-24T14:33:05.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:05.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419575+0000", 2024-04-24T14:33:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5340000000000005e-06 2024-04-24T14:33:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419616+0000", 2024-04-24T14:33:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1301999999999999e-05 2024-04-24T14:33:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419638+0000", 2024-04-24T14:33:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2177000000000001e-05 2024-04-24T14:33:05.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428578+0000", 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089398280000000004 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428665+0000", 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6644000000000006e-05 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428797+0000", 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013231500000000001 2024-04-24T14:33:05.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.429866+0000", 2024-04-24T14:33:05.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001069221 2024-04-24T14:33:05.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430684+0000", 2024-04-24T14:33:05.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081788399999999995 2024-04-24T14:33:05.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430743+0000", 2024-04-24T14:33:05.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8875999999999997e-05 2024-04-24T14:33:05.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430786+0000", 2024-04-24T14:33:05.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2398999999999998e-05 2024-04-24T14:33:05.350 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.350 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.350 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.350 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.350 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:05.350 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.706774262000003, 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:05.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:33:05.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:05.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:33:05.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:33:05.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:33:05.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:33:05.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:33:05.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:33:05.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:33:05.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:33:05.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:33:05.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:33:05.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:33:05.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:33:05.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:33:05.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:33:05.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:33:05.356 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.356 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.356 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.356 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.356 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:05.357 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.706548494000003, 2024-04-24T14:33:05.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:33:05.357 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.357 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.357 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.357 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.357 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.357 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:33:05.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.357 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:05.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:05.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:33:05.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:05.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:33:05.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:33:05.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:33:05.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:33:05.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:33:05.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:33:05.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:33:05.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:33:05.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:33:05.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:33:05.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:33:05.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:33:05.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:33:05.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:33:05.362 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.362 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.362 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.362 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.362 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:05.362 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.706371406999999, 2024-04-24T14:33:05.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:33:05.362 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.362 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:33:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:05.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:33:05.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:05.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:33:05.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:33:05.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:33:05.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:33:05.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:33:05.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:33:05.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:33:05.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:33:05.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:33:05.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:33:05.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:33:05.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:33:05.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:33:05.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:33:05.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:33:05.367 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.706267492999999, 2024-04-24T14:33:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:33:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.368 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:33:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:33:05.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:05.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:33:05.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:33:05.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:33:05.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:33:05.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:33:05.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:33:05.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:33:05.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:33:05.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:33:05.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:33:05.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:33:05.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:33:05.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:33:05.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:33:05.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:33:05.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:33:05.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:33:05.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:33:05.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:33:05.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:33:05.373 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.373 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.373 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.374 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.374 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:05.374 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.705971173000002, 2024-04-24T14:33:05.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:33:05.374 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.374 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.374 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.374 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.374 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.374 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:33:05.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.374 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:05.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:05.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:33:05.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:05.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:33:05.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:33:05.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:33:05.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:33:05.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:33:05.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:33:05.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:33:05.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:33:05.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:33:05.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:33:05.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:33:05.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:33:05.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:33:05.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:33:05.379 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.379 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.379 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.379 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.380 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:05.380 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.705879041999999, 2024-04-24T14:33:05.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:33:05.380 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.380 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.380 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.380 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.380 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.380 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:33:05.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.380 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:33:05.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:33:05.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:33:05.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:33:05.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:33:05.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:33:05.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:33:05.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:33:05.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:33:05.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:33:05.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:33:05.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:33:05.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:33:05.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:33:05.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:33:05.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:33:05.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:33:05.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.705793829999998, 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:33:05.385 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.386 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.386 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.386 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.386 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.386 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:33:05.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.386 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:05.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:05.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:33:05.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:05.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:33:05.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:33:05.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:33:05.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:33:05.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:33:05.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:33:05.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:33:05.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:33:05.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:33:05.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:33:05.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:33:05.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:33:05.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:33:05.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:33:05.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:33:05.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:33:05.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:33:05.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:33:05.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.705556948999998, 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.391 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.392 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.392 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:33:05.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.392 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:05.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:05.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:33:05.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:05.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:33:05.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:33:05.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:33:05.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:33:05.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:33:05.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:33:05.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:33:05.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:33:05.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:33:05.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:33:05.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:33:05.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:33:05.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:33:05.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:33:05.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.396 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.397 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.397 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.397 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:05.397 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.705475427000003, 2024-04-24T14:33:05.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:33:05.397 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.397 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.397 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.397 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.397 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.397 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:33:05.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.398 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:05.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:05.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:33:05.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:05.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:33:05.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:33:05.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:33:05.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:33:05.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:33:05.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:33:05.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:33:05.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:33:05.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:33:05.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:33:05.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:33:05.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:33:05.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:33:05.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:33:05.402 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.402 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.402 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.403 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.403 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:05.403 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.273756351000003, 2024-04-24T14:33:05.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01155111, 2024-04-24T14:33:05.403 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.403 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.403 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.403 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.403 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.403 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135 2024-04-24T14:33:05.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.403 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058602+0000", 2024-04-24T14:33:05.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1239999999999999e-06 2024-04-24T14:33:05.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058609+0000", 2024-04-24T14:33:05.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6719999999999998e-06 2024-04-24T14:33:05.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058626+0000", 2024-04-24T14:33:05.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7569999999999999e-05 2024-04-24T14:33:05.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065271+0000", 2024-04-24T14:33:05.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00664469 2024-04-24T14:33:05.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065398+0000", 2024-04-24T14:33:05.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012694299999999999 2024-04-24T14:33:05.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065598+0000", 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000200216 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067596+0000", 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019976450000000001 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070055+0000", 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024593710000000001 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070109+0000", 2024-04-24T14:33:05.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3690999999999998e-05 2024-04-24T14:33:05.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070147+0000", 2024-04-24T14:33:05.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7910000000000001e-05 2024-04-24T14:33:05.408 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.408 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.408 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.408 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.408 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:05.408 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.273717015999999, 2024-04-24T14:33:05.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:33:05.409 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.409 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.409 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.409 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.409 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.409 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:33:05.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.409 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:05.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:05.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:33:05.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:05.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:33:05.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:33:05.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:33:05.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:33:05.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:33:05.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:33:05.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:33:05.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:33:05.414 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.414 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.414 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.414 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.414 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:05.414 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.273675513000001, 2024-04-24T14:33:05.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:33:05.414 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.414 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.415 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.415 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.415 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.415 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:33:05.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.415 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:05.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:05.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:05.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:05.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:33:05.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:33:05.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:33:05.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:33:05.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:33:05.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:33:05.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:33:05.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:33:05.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:33:05.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:33:05.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:33:05.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:33:05.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:33:05.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:33:05.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:33:05.420 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.420 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.420 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.420 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.420 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:05.420 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.273563080000002, 2024-04-24T14:33:05.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:33:05.420 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.420 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.421 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:33:05.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.421 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:05.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:05.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:33:05.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:05.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:33:05.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:33:05.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:33:05.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:33:05.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:33:05.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:33:05.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:33:05.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:33:05.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:33:05.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:33:05.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:33:05.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:33:05.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:33:05.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:33:05.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:33:05.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:33:05.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:33:05.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:33:05.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:33:05.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:33:05.425 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.425 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.426 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.426 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.426 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:05.426 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.273534779999999, 2024-04-24T14:33:05.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:33:05.426 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.426 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.426 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:33:05.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:05.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:33:05.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:33:05.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:33:05.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:33:05.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:33:05.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:33:05.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:33:05.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:33:05.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:33:05.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:33:05.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:33:05.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:33:05.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:33:05.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:33:05.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:33:05.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:33:05.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:33:05.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:33:05.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:33:05.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:33:05.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.431 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.432 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.432 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:05.432 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.273447075999997, 2024-04-24T14:33:05.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:33:05.432 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.432 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.432 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:33:05.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.432 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:05.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:33:05.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:33:05.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:33:05.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:33:05.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:33:05.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:33:05.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:33:05.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:33:05.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:33:05.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:33:05.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:33:05.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:33:05.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:33:05.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:33:05.437 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.437 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.437 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.437 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:05.438 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:05.438 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.273324275, 2024-04-24T14:33:05.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:33:05.438 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.438 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.438 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:33:05.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.438 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:33:05.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:33:05.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:33:05.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:33:05.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:33:05.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:33:05.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:33:05.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:33:05.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:33:05.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:33:05.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:33:05.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:33:05.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:33:05.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:33:05.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:33:05.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:33:05.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:33:05.443 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.630084343, 2024-04-24T14:33:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:33:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:33:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:05.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:05.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:33:05.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:33:05.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:05.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:33:05.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:33:05.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:05.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:33:05.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:33:05.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:05.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:33:05.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:33:05.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:05.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:33:05.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:33:05.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:05.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:33:05.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:33:05.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:05.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:05.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:05.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:33:05.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:33:05.449 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.449 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.449 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.449 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:05.449 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:05.449 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:05.451 INFO:teuthology.orchestra.run.smithi090.stdout: marking collection for removal 2024-04-24T14:33:05.451 INFO:teuthology.orchestra.run.smithi090.stdout:setting '_remove' omap key 2024-04-24T14:33:05.451 INFO:teuthology.orchestra.run.smithi090.stdout:finish_remove_pgs 3.2_head removing 3.2 2024-04-24T14:33:05.451 INFO:teuthology.orchestra.run.smithi090.stdout:Remove successful 2024-04-24T14:33:05.452 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:05.452 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:05.455+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:05.520 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-24T14:33:05.551 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:05.555+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:05.557 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:05.557 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:05.558 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:05.558 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:05.567 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:05.652 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:05.655+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:05.662 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:05.662 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:05.662 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:05.662 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:05.662 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:05.671 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:05.753 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:05.755+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:05.773 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:05.774 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:05.774 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:05.774 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-24T14:33:05.774 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:05.782 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:05.854 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:05.855+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:05.879 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:05.879 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:05.879 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:05.879 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:05.888 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:05.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.2.1 2024-04-24T14:33:05.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:33:05.955 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:05.959+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:05.988 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:05.988 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:05.988 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:05.988 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:05.988 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:05.997 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:06.056 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:06.059+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:06.088 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:06.098 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:06.098 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:06.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:06.098 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-24T14:33:06.098 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:06.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:06.110 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:06.131 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:06.153 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:06.156 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:06.159+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:06.176 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:06.177 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:06.177 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:06.177 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:06.177 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:06.177 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:06.177 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:06.179 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:06.180 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:06.209 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:06.209 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:06.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:06.258 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:06.259+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:06.358 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:06.359+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:06.378 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:06.378 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:06.389 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:06.459 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:06.463+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:06.482 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:06.483 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:06.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:06.559 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:06.563+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:06.590 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:06.591 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:06.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:06.660 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:06.663+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:06.743 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:06.743 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:06.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:06.762 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:06.763+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:06.849 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:06.849 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:06.858 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:06.862 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:06.867+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:06.963 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:06.967+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:06.966 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:06.966 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:06.975 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:07.064 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:07.067+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:07.078 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:07.078 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:07.088 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:07.093 INFO:teuthology.orchestra.run.smithi090.stdout:get_pg_num_history pg_num_history pg_num_history(e31 pg_nums {1={12=1},2={15=8},3={18=4}} deleted_pools ) 2024-04-24T14:33:07.093 INFO:teuthology.orchestra.run.smithi090.stdout:Importing pgid 3.2 2024-04-24T14:33:07.093 INFO:teuthology.orchestra.run.smithi090.stdout:write_pg epoch 19 info 3.2( empty local-lis/les=18/19 n=0 ec=18/18 lis/c=18/18 les/c/f=19/19/0 sis=18) 2024-04-24T14:33:07.094 INFO:teuthology.orchestra.run.smithi090.stdout:Import successful 2024-04-24T14:33:07.165 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:07.171+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:07.190 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:07.190 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:07.198 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:07.266 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:07.267+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:07.300 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:07.300 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:07.309 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:07.369 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:07.371+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:07.455 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:07.455 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:07.464 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:07.469 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:07.471+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:07.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T14:33:07.557 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:07.557 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:07.566 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:07.611 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:07.615+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:07.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /var/log/ceph/exp.3.2.1 2024-04-24T14:33:07.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T14:33:07.670 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:07.675+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:07.713 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:07.713 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:07.722 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:07.771 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:07.775+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:07.817 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:07.817 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:07.826 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:07.872 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:07.875+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:07.929 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:33:07.929 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":35,"fsid":"6706935e-d09a-472b-b839-321319fa9e54","created":"2024-04-24T14:32:09.059925+0000","modified":"2024-04-24T14:33:05.512284+0000","last_up_change":"2024-04-24T14:33:00.483606+0000","last_in_change":"2024-04-24T14:32:11.347028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T14:32:21.233022+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T14:32:26.609964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"21","last_force_op_resend_prenautilus":"21","last_force_op_resend_preluminous":"21","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-04-24T14:32:33.987564+0000","flags":9,"flags_names":"hashpspool,incomplete_clones","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"21","last_force_op_resend_prenautilus":"21","last_force_op_resend_preluminous":"21","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":2,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a2506b4f-3486-4f51-972f-1d9e4b015346","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":20309},{"type":"v1","addr":"172.21.15.90:6817","nonce":20309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":20309},{"type":"v1","addr":"172.21.15.90:6819","nonce":20309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":20309},{"type":"v1","addr":"172.21.15.90:6823","nonce":20309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":20309},{"type":"v1","addr":"172.21.15.90:6821","nonce":20309}]},"public_addr":"172.21.15.90:6817/20309","cluster_addr":"172.21.15.90:6819/20309","heartbeat_back_addr":"172.21.15.90:6823/20309","heartbeat_front_addr":"172.21.15.90:6821/20309","state":["exists","up"]},{"osd":1,"uuid":"9e17324f-a615-494f-ba0d-7c8aa8dac95c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":18,"down_at":32,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":20308},{"type":"v1","addr":"172.21.15.90:6801","nonce":20308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":20308},{"type":"v1","addr":"172.21.15.90:6803","nonce":20308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":20308},{"type":"v1","addr":"172.21.15.90:6807","nonce":20308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":20308},{"type":"v1","addr":"172.21.15.90:6805","nonce":20308}]},"public_addr":"172.21.15.90:6801/20308","cluster_addr":"172.21.15.90:6803/20308","heartbeat_back_addr":"172.21.15.90:6807/20308","heartbeat_front_addr":"172.21.15.90:6805/20308","state":["exists"]},{"osd":2,"uuid":"f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":20310},{"type":"v1","addr":"172.21.15.90:6809","nonce":20310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":20310},{"type":"v1","addr":"172.21.15.90:6811","nonce":20310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":20310},{"type":"v1","addr":"172.21.15.90:6815","nonce":20310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":20310},{"type":"v1","addr":"172.21.15.90:6813","nonce":20310}]},"public_addr":"172.21.15.90:6809/20310","cluster_addr":"172.21.15.90:6811/20310","heartbeat_back_addr":"172.21.15.90:6815/20310","heartbeat_front_addr":"172.21.15.90:6813/20310","state":["exists","up"]},{"osd":3,"uuid":"509c16da-22b2-48e8-aa51-c320c8c4fd8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":20316},{"type":"v1","addr":"172.21.15.103:6801","nonce":20316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":20316},{"type":"v1","addr":"172.21.15.103:6803","nonce":20316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":20316},{"type":"v1","addr":"172.21.15.103:6807","nonce":20316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":20316},{"type":"v1","addr":"172.21.15.103:6805","nonce":20316}]},"public_addr":"172.21.15.103:6801/20316","cluster_addr":"172.21.15.103:6803/20316","heartbeat_back_addr":"172.21.15.103:6807/20316","heartbeat_front_addr":"172.21.15.103:6805/20316","state":["exists","up"]},{"osd":4,"uuid":"1462066b-4e59-46d4-9b85-1f61ee1d56a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":20318},{"type":"v1","addr":"172.21.15.103:6809","nonce":20318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":20318},{"type":"v1","addr":"172.21.15.103:6811","nonce":20318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":20318},{"type":"v1","addr":"172.21.15.103:6815","nonce":20318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":20318},{"type":"v1","addr":"172.21.15.103:6813","nonce":20318}]},"public_addr":"172.21.15.103:6809/20318","cluster_addr":"172.21.15.103:6811/20318","heartbeat_back_addr":"172.21.15.103:6815/20318","heartbeat_front_addr":"172.21.15.103:6813/20318","state":["exists","up"]},{"osd":5,"uuid":"4dc954a8-5f3c-4826-b9bc-98cf30519382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":20317},{"type":"v1","addr":"172.21.15.103:6817","nonce":20317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":20317},{"type":"v1","addr":"172.21.15.103:6819","nonce":20317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":20317},{"type":"v1","addr":"172.21.15.103:6823","nonce":20317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":20317},{"type":"v1","addr":"172.21.15.103:6821","nonce":20317}]},"public_addr":"172.21.15.103:6817/20317","cluster_addr":"172.21.15.103:6819/20317","heartbeat_back_addr":"172.21.15.103:6823/20317","heartbeat_front_addr":"172.21.15.103:6821/20317","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.792899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.580414+0000","dead_epoch":32},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.713336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.579638+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.570510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.736558+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T14:33:07.944 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T14:32:21.233022+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T14:32:26.609964+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '21', 'last_force_op_resend_prenautilus': '21', 'last_force_op_resend_preluminous': '21', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-04-24T14:32:33.987564+0000', 'flags': 9, 'flags_names': 'hashpspool,incomplete_clones', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '21', 'last_force_op_resend_prenautilus': '21', 'last_force_op_resend_preluminous': '21', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 2, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2024-04-24T14:33:07.944 DEBUG:teuthology.orchestra.run.smithi090:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op apply-layout-settings --pool .mgr 2024-04-24T14:33:07.967 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:07.967 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:07.973 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:07.975+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:07.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:08.074 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:08.075+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:08.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:08.175 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:08.179+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:08.178 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:08.178 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:08.181 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:33:08.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:08.276 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:08.283+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:08.284 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:08.284 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:08.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:08.302 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:08.325 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:08.346 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:08.367 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:08.377 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:08.379+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:08.390 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:08.390 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:08.391 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:08.391 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:08.391 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:08.391 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:08.391 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:08.393 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:08.394 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:08.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:08.477 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:08.479+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:08.535 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-24T14:33:08.547 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:08.547 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:08.547 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:08.548 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:08.548 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:08.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:08.578 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:08.579+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:08.652 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-04-24T14:33:08.652 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:08.661 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:08.702 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:08.707+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:08.771 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:08.771 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:08.771 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:08.771 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:08.779 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:08.783+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:08.781 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:08.879 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:08.883+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:08.880 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:08.880 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:08.880 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:08.880 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:08.880 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:08.889 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:08.981 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:08.983+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:08.995 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:08.995 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:08.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:08.995 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:08.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:08.995 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:08.996 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:08.996 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.67246695, 2024-04-24T14:33:08.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011830553000000001, 2024-04-24T14:33:08.996 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:08.996 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:08.996 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:08.996 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:08.996 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:08.996 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 36 2024-04-24T14:33:08.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:08.996 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:08.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:08.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:08.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:08.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:08.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:08.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:08.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:08.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:08.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:08.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:08.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:08.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:08.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326000+0000", 2024-04-24T14:33:08.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:08.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:08.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:08.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:08.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326006+0000", 2024-04-24T14:33:08.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9340000000000003e-06 2024-04-24T14:33:08.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:08.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:08.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:08.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326044+0000", 2024-04-24T14:33:08.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8124999999999998e-05 2024-04-24T14:33:08.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:08.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:08.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:08.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326054+0000", 2024-04-24T14:33:08.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9650000000000003e-06 2024-04-24T14:33:08.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:08.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:08.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:08.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332746+0000", 2024-04-24T14:33:08.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006692539 2024-04-24T14:33:08.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:08.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:08.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:08.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332833+0000", 2024-04-24T14:33:08.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6482000000000003e-05 2024-04-24T14:33:08.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:08.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:08.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:08.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332947+0000", 2024-04-24T14:33:08.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114472 2024-04-24T14:33:09.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334320+0000", 2024-04-24T14:33:09.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001372489 2024-04-24T14:33:09.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337717+0000", 2024-04-24T14:33:09.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003396849 2024-04-24T14:33:09.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337789+0000", 2024-04-24T14:33:09.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2080000000000001e-05 2024-04-24T14:33:09.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337831+0000", 2024-04-24T14:33:09.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.244e-05 2024-04-24T14:33:09.001 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.001 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.001 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.001 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.001 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:09.002 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.672405087999998, 2024-04-24T14:33:09.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:33:09.002 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.002 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.002 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.002 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.002 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.002 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:33:09.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.002 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:09.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:33:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:33:09.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:33:09.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:33:09.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:33:09.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:33:09.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:33:09.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:33:09.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:33:09.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:33:09.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:33:09.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:33:09.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:33:09.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:33:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:33:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:33:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:33:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:33:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:33:09.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:33:09.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:33:09.007 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.007 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.007 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.007 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.007 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:09.007 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.578900615000002, 2024-04-24T14:33:09.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011218514000000001, 2024-04-24T14:33:09.007 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 88 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:09.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.009 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419565+0000", 2024-04-24T14:33:09.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:09.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.009 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419575+0000", 2024-04-24T14:33:09.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5340000000000005e-06 2024-04-24T14:33:09.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419616+0000", 2024-04-24T14:33:09.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1301999999999999e-05 2024-04-24T14:33:09.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419638+0000", 2024-04-24T14:33:09.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2177000000000001e-05 2024-04-24T14:33:09.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428578+0000", 2024-04-24T14:33:09.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089398280000000004 2024-04-24T14:33:09.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428665+0000", 2024-04-24T14:33:09.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6644000000000006e-05 2024-04-24T14:33:09.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428797+0000", 2024-04-24T14:33:09.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013231500000000001 2024-04-24T14:33:09.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.429866+0000", 2024-04-24T14:33:09.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001069221 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430684+0000", 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081788399999999995 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430743+0000", 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8875999999999997e-05 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430786+0000", 2024-04-24T14:33:09.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2398999999999998e-05 2024-04-24T14:33:09.013 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.013 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.013 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.013 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.013 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:09.013 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.372890146000003, 2024-04-24T14:33:09.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:33:09.013 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.013 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.013 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.013 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.013 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.014 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:33:09.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.014 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:09.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:09.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:33:09.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:33:09.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:33:09.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:33:09.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:33:09.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:33:09.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:33:09.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:33:09.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:33:09.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:33:09.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:33:09.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:33:09.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:33:09.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:33:09.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:33:09.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:33:09.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:33:09.018 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.018 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.018 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.372664378000003, 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:09.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:09.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:33:09.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:09.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:33:09.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:33:09.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:33:09.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:33:09.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:33:09.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:33:09.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:33:09.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:33:09.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:33:09.024 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.024 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.024 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.024 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.024 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:09.024 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.372487290999999, 2024-04-24T14:33:09.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:33:09.025 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.025 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.025 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.025 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.025 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.025 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:33:09.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.025 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:09.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:09.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:33:09.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:09.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:33:09.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:33:09.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:33:09.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:33:09.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:33:09.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:33:09.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:33:09.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:33:09.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:33:09.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:33:09.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:33:09.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:33:09.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:33:09.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:33:09.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:33:09.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:33:09.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:33:09.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:33:09.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:33:09.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:33:09.030 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.030 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.030 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.030 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.030 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:09.030 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.372383376999998, 2024-04-24T14:33:09.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:33:09.030 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.030 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.031 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.031 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.031 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.031 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:33:09.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.031 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:09.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:09.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:33:09.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:09.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:33:09.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:33:09.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:33:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:33:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:33:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:33:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:33:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:33:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:33:09.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:33:09.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:33:09.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:33:09.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:33:09.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:33:09.036 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.036 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.036 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.036 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.036 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:09.036 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.372087057000002, 2024-04-24T14:33:09.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:33:09.037 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.037 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.037 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.037 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.037 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.037 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:33:09.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.037 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:09.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:09.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:33:09.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:09.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:33:09.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:33:09.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:33:09.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:33:09.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:33:09.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:33:09.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:33:09.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:33:09.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:33:09.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:33:09.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:33:09.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:33:09.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:33:09.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:33:09.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.042 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:33:09.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:33:09.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.042 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:33:09.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:33:09.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:33:09.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:33:09.043 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.043 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.043 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.043 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.043 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:09.044 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.371994925999999, 2024-04-24T14:33:09.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:33:09.044 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.044 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.044 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.044 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.044 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.044 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:33:09.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.044 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:09.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:09.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:33:09.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:09.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:33:09.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:33:09.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:33:09.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:33:09.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:33:09.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:33:09.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:33:09.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:33:09.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:33:09.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:33:09.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:33:09.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:33:09.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:33:09.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:33:09.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:33:09.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:33:09.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:33:09.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:33:09.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:33:09.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:33:09.051 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.051 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.052 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.052 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.052 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:09.052 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.371909713999997, 2024-04-24T14:33:09.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:33:09.052 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.052 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.052 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.053 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.053 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.053 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:33:09.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.053 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:09.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:09.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:33:09.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:09.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:33:09.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:33:09.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:33:09.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:33:09.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:33:09.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:33:09.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:33:09.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:33:09.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:33:09.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:33:09.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:33:09.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:33:09.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:33:09.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:33:09.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:33:09.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:33:09.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:33:09.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:33:09.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:33:09.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:33:09.058 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.059 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.059 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.059 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.059 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:09.059 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.371672832999998, 2024-04-24T14:33:09.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:33:09.059 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.059 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.059 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.059 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.060 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.060 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:33:09.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.060 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:09.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:09.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:33:09.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:09.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:33:09.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:33:09.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:33:09.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:33:09.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:33:09.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:33:09.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:33:09.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:33:09.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:33:09.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:33:09.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:33:09.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:33:09.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:33:09.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:33:09.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:33:09.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:33:09.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:33:09.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:33:09.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:33:09.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:33:09.065 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.065 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.065 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.066 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.066 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:09.066 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.371591311000003, 2024-04-24T14:33:09.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:33:09.066 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.066 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.066 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.066 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.066 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.066 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:33:09.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.067 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:09.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:09.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:33:09.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:09.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:33:09.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:33:09.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:33:09.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:33:09.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:33:09.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:33:09.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:33:09.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:33:09.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:33:09.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:33:09.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:33:09.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:33:09.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:33:09.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:33:09.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:33:09.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:33:09.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:33:09.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:33:09.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:33:09.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:33:09.072 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.073 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.073 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.073 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.073 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:09.073 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.939872235000003, 2024-04-24T14:33:09.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01155111, 2024-04-24T14:33:09.073 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.073 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.073 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.074 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.074 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.074 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135 2024-04-24T14:33:09.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.074 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:09.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:09.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:09.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:09.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058602+0000", 2024-04-24T14:33:09.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1239999999999999e-06 2024-04-24T14:33:09.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058609+0000", 2024-04-24T14:33:09.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6719999999999998e-06 2024-04-24T14:33:09.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058626+0000", 2024-04-24T14:33:09.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7569999999999999e-05 2024-04-24T14:33:09.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065271+0000", 2024-04-24T14:33:09.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00664469 2024-04-24T14:33:09.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065398+0000", 2024-04-24T14:33:09.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012694299999999999 2024-04-24T14:33:09.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065598+0000", 2024-04-24T14:33:09.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000200216 2024-04-24T14:33:09.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067596+0000", 2024-04-24T14:33:09.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019976450000000001 2024-04-24T14:33:09.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070055+0000", 2024-04-24T14:33:09.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024593710000000001 2024-04-24T14:33:09.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070109+0000", 2024-04-24T14:33:09.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3690999999999998e-05 2024-04-24T14:33:09.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070147+0000", 2024-04-24T14:33:09.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7910000000000001e-05 2024-04-24T14:33:09.080 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.080 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.080 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.080 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.080 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:09.080 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.939832899999999, 2024-04-24T14:33:09.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:33:09.080 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.080 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.081 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.081 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.081 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.081 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:33:09.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.081 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:09.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:09.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:33:09.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:09.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:33:09.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:33:09.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:33:09.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:33:09.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:33:09.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:33:09.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:33:09.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:33:09.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:33:09.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:33:09.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:33:09.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:33:09.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:33:09.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:33:09.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:33:09.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:33:09.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:33:09.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:33:09.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:33:09.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:33:09.087 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.087 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.087 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.087 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.087 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:09.087 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.939791397, 2024-04-24T14:33:09.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:33:09.087 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.087 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.087 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.088 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.088 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.088 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:33:09.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.088 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:09.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:33:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:33:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:33:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:33:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:33:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:33:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:33:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:33:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:33:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:33:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:33:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:33:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:33:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:33:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:33:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:33:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:33:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:33:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:33:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:33:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.939678964000002, 2024-04-24T14:33:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:33:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:33:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:33:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:33:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:33:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:33:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:33:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:33:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:33:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:33:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:33:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:33:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:33:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:33:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:33:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:33:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:33:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:33:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:33:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:33:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:33:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:33:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:33:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.939650663999998, 2024-04-24T14:33:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:33:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:33:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:33:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:33:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:33:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:33:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:33:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:33:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:33:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:33:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:33:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:33:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:33:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:33:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:33:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:33:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:33:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:33:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:33:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:33:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:33:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:33:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:33:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.939562960000003, 2024-04-24T14:33:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:33:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:33:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:33:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:33:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:33:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:33:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:33:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:33:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:33:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:33:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:33:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:33:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:33:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:33:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:33:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:33:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:33:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:33:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:33:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:33:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:33:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:33:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:33:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.939440159, 2024-04-24T14:33:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:33:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:33:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:33:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:33:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:33:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:33:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:33:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:33:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:33:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:33:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:33:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:33:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:33:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:33:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:33:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:33:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:33:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:33:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:33:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:33:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:33:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:33:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:33:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:33:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.296200227, 2024-04-24T14:33:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:33:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:33:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:33:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:33:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:33:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:33:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:33:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:33:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:33:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:33:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:33:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:33:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:33:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:33:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:33:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:33:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:33:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:09.127 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:09.128 INFO:teuthology.orchestra.run.smithi090.stderr:Nothing to do for non-filestore backend 2024-04-24T14:33:09.129 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:09.129 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:09.131+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:09.229 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:09.231+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:09.278 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:09.278 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:09.278 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:09.278 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:09.287 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:09.329 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:09.335+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:09.383 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:09.383 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:09.383 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:09.383 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:09.383 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:09.392 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:09.430 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:09.431+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:09.489 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:09.489 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:09.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:09.489 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-24T14:33:09.489 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:09.498 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:09.530 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:09.535+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:09.601 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:09.601 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:09.601 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:09.601 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:09.610 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:09.622 DEBUG:teuthology.orchestra.run.smithi090:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op apply-layout-settings --pool rbd 2024-04-24T14:33:09.631 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:09.635+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:09.706 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:09.707 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:09.707 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:09.707 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:09.707 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:09.715 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:09.732 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:09.735+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:09.815 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:09.815 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:09.815 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:09.815 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-24T14:33:09.815 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:09.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:09.833 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:09.835+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:09.931 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:09.931 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:09.934 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:09.935+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:09.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:10.034 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:10.039+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:10.053 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:10.053 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:10.053 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:10.054 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:10.054 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:10.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:10.136 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:10.139+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:10.167 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:10.167 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:10.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:10.167 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:10.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.167 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:36 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'22, mlcod=13'17)", 2024-04-24T14:33:10.168 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:10.168 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.839233639, 2024-04-24T14:33:10.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043791400000000001, 2024-04-24T14:33:10.168 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.168 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.168 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:10.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:10.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331851+0000", 2024-04-24T14:33:10.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331864+0000", 2024-04-24T14:33:10.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2767e-05 2024-04-24T14:33:10.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331881+0000", 2024-04-24T14:33:10.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6586e-05 2024-04-24T14:33:10.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331894+0000", 2024-04-24T14:33:10.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2976000000000001e-05 2024-04-24T14:33:10.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332161+0000", 2024-04-24T14:33:10.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026765399999999998 2024-04-24T14:33:10.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332197+0000", 2024-04-24T14:33:10.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5565999999999998e-05 2024-04-24T14:33:10.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336158+0000", 2024-04-24T14:33:10.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039616820000000002 2024-04-24T14:33:10.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336232+0000", 2024-04-24T14:33:10.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3151999999999994e-05 2024-04-24T14:33:10.172 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.172 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.172 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.172 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:37 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'23, mlcod=13'18)", 2024-04-24T14:33:10.172 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:10.172 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.838347161000002, 2024-04-24T14:33:10.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040417389999999999, 2024-04-24T14:33:10.172 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.172 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.172 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332738+0000", 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332750+0000", 2024-04-24T14:33:10.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2432000000000001e-05 2024-04-24T14:33:10.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332768+0000", 2024-04-24T14:33:10.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7314000000000001e-05 2024-04-24T14:33:10.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332780+0000", 2024-04-24T14:33:10.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2716e-05 2024-04-24T14:33:10.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332925+0000", 2024-04-24T14:33:10.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144587 2024-04-24T14:33:10.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332955+0000", 2024-04-24T14:33:10.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0246000000000001e-05 2024-04-24T14:33:10.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336698+0000", 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037430559999999998 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336781+0000", 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2447999999999994e-05 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:39 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'25, mlcod=13'22)", 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.832228837999999, 2024-04-24T14:33:10.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038736740000000001, 2024-04-24T14:33:10.177 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.177 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.177 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:10.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:10.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338856+0000", 2024-04-24T14:33:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338995+0000", 2024-04-24T14:33:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013868100000000001 2024-04-24T14:33:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339029+0000", 2024-04-24T14:33:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4372000000000002e-05 2024-04-24T14:33:10.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339050+0000", 2024-04-24T14:33:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0656999999999999e-05 2024-04-24T14:33:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339156+0000", 2024-04-24T14:33:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106177 2024-04-24T14:33:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339188+0000", 2024-04-24T14:33:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.243e-05 2024-04-24T14:33:10.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342648+0000", 2024-04-24T14:33:10.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034601689999999999 2024-04-24T14:33:10.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342731+0000", 2024-04-24T14:33:10.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2534999999999995e-05 2024-04-24T14:33:10.180 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.180 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.180 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.181 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:67 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'41, mlcod=13'39)", 2024-04-24T14:33:10.181 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:10.181 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.781492554000003, 2024-04-24T14:33:10.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031029009999999999, 2024-04-24T14:33:10.181 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.181 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.181 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:10.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:10.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389592+0000", 2024-04-24T14:33:10.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389603+0000", 2024-04-24T14:33:10.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090000000000001e-05 2024-04-24T14:33:10.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389631+0000", 2024-04-24T14:33:10.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7772000000000001e-05 2024-04-24T14:33:10.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389646+0000", 2024-04-24T14:33:10.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.432e-05 2024-04-24T14:33:10.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389732+0000", 2024-04-24T14:33:10.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6331999999999999e-05 2024-04-24T14:33:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389771+0000", 2024-04-24T14:33:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8751999999999998e-05 2024-04-24T14:33:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392585+0000", 2024-04-24T14:33:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028147469999999998 2024-04-24T14:33:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392696+0000", 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111121 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.780349942999997, 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030475519999999998, 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:10.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:10.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390735+0000", 2024-04-24T14:33:10.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390804+0000", 2024-04-24T14:33:10.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9488999999999998e-05 2024-04-24T14:33:10.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390832+0000", 2024-04-24T14:33:10.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7946e-05 2024-04-24T14:33:10.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390847+0000", 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4693e-05 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391508+0000", 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066154799999999998 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391538+0000", 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0029e-05 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393698+0000", 2024-04-24T14:33:10.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002160052 2024-04-24T14:33:10.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393784+0000", 2024-04-24T14:33:10.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5414999999999995e-05 2024-04-24T14:33:10.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.189 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.189 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:72 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'46, mlcod=13'43)", 2024-04-24T14:33:10.189 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:10.189 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.774176449999999, 2024-04-24T14:33:10.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036059460000000001, 2024-04-24T14:33:10.190 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.190 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.190 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:10.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:10.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396908+0000", 2024-04-24T14:33:10.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396968+0000", 2024-04-24T14:33:10.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9412e-05 2024-04-24T14:33:10.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397010+0000", 2024-04-24T14:33:10.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2169000000000002e-05 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397022+0000", 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1749999999999999e-05 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397096+0000", 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4701000000000002e-05 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397126+0000", 2024-04-24T14:33:10.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2024-04-24T14:33:10.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400443+0000", 2024-04-24T14:33:10.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033172140000000002 2024-04-24T14:33:10.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400516+0000", 2024-04-24T14:33:10.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2181999999999994e-05 2024-04-24T14:33:10.193 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.193 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.193 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.194 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:73 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'47, mlcod=13'43)", 2024-04-24T14:33:10.194 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:10.194 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.773329046000001, 2024-04-24T14:33:10.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055152170000000002, 2024-04-24T14:33:10.194 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.194 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.194 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:10.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397756+0000", 2024-04-24T14:33:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397884+0000", 2024-04-24T14:33:10.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012842100000000001 2024-04-24T14:33:10.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397923+0000", 2024-04-24T14:33:10.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8942000000000001e-05 2024-04-24T14:33:10.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397939+0000", 2024-04-24T14:33:10.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6073e-05 2024-04-24T14:33:10.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398106+0000", 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016737100000000001 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398137+0000", 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0496000000000001e-05 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403191+0000", 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00505391 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403272+0000", 2024-04-24T14:33:10.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1483999999999994e-05 2024-04-24T14:33:10.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.198 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.198 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:74 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'48, mlcod=13'43)", 2024-04-24T14:33:10.198 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:10.198 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.772530019999998, 2024-04-24T14:33:10.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048091760000000001, 2024-04-24T14:33:10.198 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.198 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.198 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398555+0000", 2024-04-24T14:33:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398570+0000", 2024-04-24T14:33:10.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.558e-05 2024-04-24T14:33:10.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398610+0000", 2024-04-24T14:33:10.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0028999999999999e-05 2024-04-24T14:33:10.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398625+0000", 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398701+0000", 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6507000000000005e-05 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398731+0000", 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9356000000000002e-05 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403281+0000", 2024-04-24T14:33:10.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004549796 2024-04-24T14:33:10.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403365+0000", 2024-04-24T14:33:10.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4685000000000007e-05 2024-04-24T14:33:10.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.202 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.202 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-04-24T14:33:10.202 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:10.202 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.751217095000001, 2024-04-24T14:33:10.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038140460000000002, 2024-04-24T14:33:10.203 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.203 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.203 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:10.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:10.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419868+0000", 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419882+0000", 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.45e-05 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419919+0000", 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6939e-05 2024-04-24T14:33:10.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419961+0000", 2024-04-24T14:33:10.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2148999999999999e-05 2024-04-24T14:33:10.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420084+0000", 2024-04-24T14:33:10.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122958 2024-04-24T14:33:10.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420122+0000", 2024-04-24T14:33:10.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8334999999999998e-05 2024-04-24T14:33:10.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423616+0000", 2024-04-24T14:33:10.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003493566 2024-04-24T14:33:10.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423683+0000", 2024-04-24T14:33:10.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7149000000000004e-05 2024-04-24T14:33:10.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.206 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.749805340000002, 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030290730000000002, 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421279+0000", 2024-04-24T14:33:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421352+0000", 2024-04-24T14:33:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2251999999999999e-05 2024-04-24T14:33:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421378+0000", 2024-04-24T14:33:10.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6006000000000001e-05 2024-04-24T14:33:10.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421388+0000", 2024-04-24T14:33:10.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0373e-05 2024-04-24T14:33:10.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421473+0000", 2024-04-24T14:33:10.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4665000000000003e-05 2024-04-24T14:33:10.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421512+0000", 2024-04-24T14:33:10.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9619000000000001e-05 2024-04-24T14:33:10.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424198+0000", 2024-04-24T14:33:10.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026854610000000001 2024-04-24T14:33:10.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424310+0000", 2024-04-24T14:33:10.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011208 2024-04-24T14:33:10.211 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.211 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.211 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.211 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:87 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'57, mlcod=13'51)", 2024-04-24T14:33:10.211 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:10.211 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.745542381, 2024-04-24T14:33:10.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002974057, 2024-04-24T14:33:10.211 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.211 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425542+0000", 2024-04-24T14:33:10.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425614+0000", 2024-04-24T14:33:10.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1328999999999994e-05 2024-04-24T14:33:10.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425664+0000", 2024-04-24T14:33:10.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0491999999999997e-05 2024-04-24T14:33:10.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425676+0000", 2024-04-24T14:33:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.134e-05 2024-04-24T14:33:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425798+0000", 2024-04-24T14:33:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012221700000000001 2024-04-24T14:33:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425860+0000", 2024-04-24T14:33:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1761999999999996e-05 2024-04-24T14:33:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428368+0000", 2024-04-24T14:33:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025084 2024-04-24T14:33:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428518+0000", 2024-04-24T14:33:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149884 2024-04-24T14:33:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-04-24T14:33:10.215 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:10.216 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.545433656, 2024-04-24T14:33:10.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002919694, 2024-04-24T14:33:10.216 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.216 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.216 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:10.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:10.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625651+0000", 2024-04-24T14:33:10.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:10.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625667+0000", 2024-04-24T14:33:10.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5973000000000001e-05 2024-04-24T14:33:10.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625689+0000", 2024-04-24T14:33:10.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:33:10.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625705+0000", 2024-04-24T14:33:10.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5797000000000001e-05 2024-04-24T14:33:10.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625805+0000", 2024-04-24T14:33:10.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9238000000000001e-05 2024-04-24T14:33:10.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625837+0000", 2024-04-24T14:33:10.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2722000000000003e-05 2024-04-24T14:33:10.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628476+0000", 2024-04-24T14:33:10.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026391980000000002 2024-04-24T14:33:10.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628572+0000", 2024-04-24T14:33:10.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5754000000000005e-05 2024-04-24T14:33:10.219 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.220 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.220 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'88)", 2024-04-24T14:33:10.220 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:10.220 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.105495777999998, 2024-04-24T14:33:10.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031994179999999999, 2024-04-24T14:33:10.220 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.220 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.220 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:10.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:10.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065589+0000", 2024-04-24T14:33:10.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065602+0000", 2024-04-24T14:33:10.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2737e-05 2024-04-24T14:33:10.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065620+0000", 2024-04-24T14:33:10.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8546000000000002e-05 2024-04-24T14:33:10.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065629+0000", 2024-04-24T14:33:10.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1600000000000004e-06 2024-04-24T14:33:10.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065729+0000", 2024-04-24T14:33:10.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9933999999999996e-05 2024-04-24T14:33:10.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065792+0000", 2024-04-24T14:33:10.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2712000000000005e-05 2024-04-24T14:33:10.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068667+0000", 2024-04-24T14:33:10.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028752920000000002 2024-04-24T14:33:10.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068790+0000", 2024-04-24T14:33:10.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012244400000000001 2024-04-24T14:33:10.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.224 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.224 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:137 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'91)", 2024-04-24T14:33:10.224 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:10.224 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.104140041999997, 2024-04-24T14:33:10.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052524399999999997, 2024-04-24T14:33:10.225 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.225 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.225 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:10.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:10.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066945+0000", 2024-04-24T14:33:10.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067031+0000", 2024-04-24T14:33:10.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6228000000000002e-05 2024-04-24T14:33:10.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067060+0000", 2024-04-24T14:33:10.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8445999999999998e-05 2024-04-24T14:33:10.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067069+0000", 2024-04-24T14:33:10.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999996e-06 2024-04-24T14:33:10.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067172+0000", 2024-04-24T14:33:10.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010285099999999999 2024-04-24T14:33:10.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067234+0000", 2024-04-24T14:33:10.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2068000000000003e-05 2024-04-24T14:33:10.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072090+0000", 2024-04-24T14:33:10.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048560859999999999 2024-04-24T14:33:10.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072199+0000", 2024-04-24T14:33:10.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108384 2024-04-24T14:33:10.228 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.228 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.228 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.229 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:138 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'96, mlcod=13'91)", 2024-04-24T14:33:10.229 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:10.229 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.103140277999998, 2024-04-24T14:33:10.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053159380000000001, 2024-04-24T14:33:10.229 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.229 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.229 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:10.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:10.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067945+0000", 2024-04-24T14:33:10.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067957+0000", 2024-04-24T14:33:10.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1963e-05 2024-04-24T14:33:10.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067982+0000", 2024-04-24T14:33:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5570000000000001e-05 2024-04-24T14:33:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067994+0000", 2024-04-24T14:33:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2306e-05 2024-04-24T14:33:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068282+0000", 2024-04-24T14:33:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028799000000000001 2024-04-24T14:33:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068307+0000", 2024-04-24T14:33:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4351000000000001e-05 2024-04-24T14:33:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073142+0000", 2024-04-24T14:33:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004835326 2024-04-24T14:33:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073262+0000", 2024-04-24T14:33:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119695 2024-04-24T14:33:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:139 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T14:33:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.101657451000001, 2024-04-24T14:33:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039447479999999997, 2024-04-24T14:33:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:10.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:10.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069428+0000", 2024-04-24T14:33:10.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:10.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069440+0000", 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2047e-05 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069469+0000", 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8852000000000001e-05 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069477+0000", 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-04-24T14:33:10.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069531+0000", 2024-04-24T14:33:10.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4126999999999999e-05 2024-04-24T14:33:10.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069559+0000", 2024-04-24T14:33:10.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7881e-05 2024-04-24T14:33:10.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073270+0000", 2024-04-24T14:33:10.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037104519999999999 2024-04-24T14:33:10.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073373+0000", 2024-04-24T14:33:10.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010380899999999999 2024-04-24T14:33:10.237 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.237 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.237 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.237 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:140 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'92)", 2024-04-24T14:33:10.237 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:10.237 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.100831575000001, 2024-04-24T14:33:10.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031722510000000001, 2024-04-24T14:33:10.238 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.238 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.238 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:10.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:10.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070253+0000", 2024-04-24T14:33:10.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:10.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070307+0000", 2024-04-24T14:33:10.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3909000000000002e-05 2024-04-24T14:33:10.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070330+0000", 2024-04-24T14:33:10.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2599000000000002e-05 2024-04-24T14:33:10.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070337+0000", 2024-04-24T14:33:10.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3030000000000002e-06 2024-04-24T14:33:10.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070377+0000", 2024-04-24T14:33:10.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9916000000000002e-05 2024-04-24T14:33:10.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070398+0000", 2024-04-24T14:33:10.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0579000000000001e-05 2024-04-24T14:33:10.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073377+0000", 2024-04-24T14:33:10.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002979013 2024-04-24T14:33:10.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073427+0000", 2024-04-24T14:33:10.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0078999999999999e-05 2024-04-24T14:33:10.241 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.241 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.241 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.242 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:143 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'101, mlcod=13'98)", 2024-04-24T14:33:10.242 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:10.242 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.094646294999997, 2024-04-24T14:33:10.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030247590000000001, 2024-04-24T14:33:10.242 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.242 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.242 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:10.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:10.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076438+0000", 2024-04-24T14:33:10.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076533+0000", 2024-04-24T14:33:10.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4443000000000004e-05 2024-04-24T14:33:10.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076563+0000", 2024-04-24T14:33:10.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0386999999999999e-05 2024-04-24T14:33:10.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076604+0000", 2024-04-24T14:33:10.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1012e-05 2024-04-24T14:33:10.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076711+0000", 2024-04-24T14:33:10.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010637300000000001 2024-04-24T14:33:10.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076737+0000", 2024-04-24T14:33:10.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6190000000000002e-05 2024-04-24T14:33:10.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079368+0000", 2024-04-24T14:33:10.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026313529999999999 2024-04-24T14:33:10.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079465+0000", 2024-04-24T14:33:10.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6426000000000001e-05 2024-04-24T14:33:10.246 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.246 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.246 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.246 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:159 1.0 e14/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'113, mlcod=14'109)", 2024-04-24T14:33:10.246 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:10.246 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.460826189000002, 2024-04-24T14:33:10.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052053589999999997, 2024-04-24T14:33:10.246 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.247 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.247 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:10.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:10.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710259+0000", 2024-04-24T14:33:10.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:10.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710272+0000", 2024-04-24T14:33:10.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.382e-05 2024-04-24T14:33:10.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710303+0000", 2024-04-24T14:33:10.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0629000000000001e-05 2024-04-24T14:33:10.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710322+0000", 2024-04-24T14:33:10.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8859999999999999e-05 2024-04-24T14:33:10.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710479+0000", 2024-04-24T14:33:10.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157076 2024-04-24T14:33:10.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710535+0000", 2024-04-24T14:33:10.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6289000000000003e-05 2024-04-24T14:33:10.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715391+0000", 2024-04-24T14:33:10.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048557139999999997 2024-04-24T14:33:10.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715465+0000", 2024-04-24T14:33:10.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4301000000000006e-05 2024-04-24T14:33:10.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.250 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.251 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:161 1.0 e14/12 1:30306672:devicehealth::main.db.0000000000000000:head v 14'115, mlcod=14'112)", 2024-04-24T14:33:10.251 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:10.251 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.453715387000003, 2024-04-24T14:33:10.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039960450000000002, 2024-04-24T14:33:10.251 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:10.251 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:10.251 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:10.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:10.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:10.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717370+0000", 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717499+0000", 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012877400000000001 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:10.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717513+0000", 2024-04-24T14:33:10.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4229999999999999e-05 2024-04-24T14:33:10.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:10.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717521+0000", 2024-04-24T14:33:10.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2024-04-24T14:33:10.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:10.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718040+0000", 2024-04-24T14:33:10.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051920099999999995 2024-04-24T14:33:10.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718094+0000", 2024-04-24T14:33:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859000000000001e-05 2024-04-24T14:33:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721278+0000", 2024-04-24T14:33:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031838560000000001 2024-04-24T14:33:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721367+0000", 2024-04-24T14:33:10.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9241999999999997e-05 2024-04-24T14:33:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:10.255 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:10.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:10.257 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:10.259+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:10.357 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:10.359+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:10.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:33:10.406 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:10.407 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:10.407 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:10.407 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:10.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:10.462 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:10.467+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:10.524 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:10.542 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:10.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T14:33:10.559 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:10.563+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:10.560 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:10.560 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:10.565 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:10.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:10.587 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:10.608 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:10.608 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:10.608 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:10.608 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:10.608 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:10.608 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:10.608 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:10.611 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:10.611 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:10.658 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:10.659 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:10.660 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:10.663+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:10.668 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:10.761 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:10.763+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:10.775 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:10.775 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:10.784 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:10.824 INFO:teuthology.orchestra.run.smithi090.stderr:Nothing to do for non-filestore backend 2024-04-24T14:33:10.861 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:10.863+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:10.879 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:10.879 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:10.889 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:10.962 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:10.963+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:10.986 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:10.986 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:10.995 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:11.063 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:11.067+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:11.148 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:11.148 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:11.157 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:11.163 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:11.167+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:11.257 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:11.257 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:11.264 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:11.267+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:11.266 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:11.362 DEBUG:teuthology.orchestra.run.smithi090:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op apply-layout-settings --pool cache 2024-04-24T14:33:11.365 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:11.367+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:11.365 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:11.365 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:11.374 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:11.465 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:11.467+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:11.521 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:11.521 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:11.531 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:11.546 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-24T14:33:11.566 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:11.571+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:11.623 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:11.623 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:11.632 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:11.668 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:11.671+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:11.729 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:11.729 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:11.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:11.769 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:11.771+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:11.843 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:11.844 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:11.855 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:11.870 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:11.871+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:11.953 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:11.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:11.971 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:11.974+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:12.061 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:12.061 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:12.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:12.072 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:12.075+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:12.169 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:12.169 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:12.173 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:12.174+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:12.178 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:12.274 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:12.274+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:12.290 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:12.290 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:12.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:12.375 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:12.378+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:12.399 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:12.399 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:12.408 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:12.476 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:12.479+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:12.516 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:12.517 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:12.526 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:12.550 INFO:teuthology.orchestra.run.smithi090.stderr:Nothing to do for non-filestore backend 2024-04-24T14:33:12.577 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:12.578+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:12.612 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:33:12.627 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:12.627 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:12.636 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:12.678 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:12.683+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:12.735 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:12.735 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:12.744 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:12.755 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:12.778 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:12.779 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:12.782+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:12.801 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:12.824 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:12.844 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:12.844 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:12.844 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:12.844 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:12.846 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:12.846 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:12.847 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:12.847 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:12.847 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:12.847 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:12.847 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:12.851 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:12.851 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:12.853 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:12.894 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:12.899+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:12.980 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:12.983+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:12.996 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:12.996 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:12.997 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:12.997 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:12.997 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:13.006 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:13.081 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:13.082+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:13.106 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:13.106 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:13.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:13.106 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [] 2024-04-24T14:33:13.106 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:13.115 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:13.181 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:13.182+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:13.261 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:13.261 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:13.261 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:13.261 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:13.270 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:13.315 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:13.318+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:13.367 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:13.367 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:13.367 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:13.367 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:13.367 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:13.376 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:13.383 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:13.386+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:13.472 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:1 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.27662048, 2024-04-24T14:33:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002159392, 2024-04-24T14:33:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 1 2024-04-24T14:33:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201710+0000", 2024-04-24T14:33:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:33:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201731+0000", 2024-04-24T14:33:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0835999999999998e-05 2024-04-24T14:33:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201769+0000", 2024-04-24T14:33:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7472999999999998e-05 2024-04-24T14:33:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201796+0000", 2024-04-24T14:33:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6798999999999999e-05 2024-04-24T14:33:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201923+0000", 2024-04-24T14:33:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012770699999999999 2024-04-24T14:33:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203873+0000", 2024-04-24T14:33:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00194992 2024-04-24T14:33:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.27505474299999999, 2024-04-24T14:33:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026872300000000001, 2024-04-24T14:33:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203277+0000", 2024-04-24T14:33:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203292+0000", 2024-04-24T14:33:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4277000000000001e-05 2024-04-24T14:33:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203327+0000", 2024-04-24T14:33:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5219000000000003e-05 2024-04-24T14:33:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203350+0000", 2024-04-24T14:33:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2807e-05 2024-04-24T14:33:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203437+0000", 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7070999999999994e-05 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203548+0000", 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000111576 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:2 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.27494143799999998, 2024-04-24T14:33:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047788300000000002, 2024-04-24T14:33:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203391+0000", 2024-04-24T14:33:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203400+0000", 2024-04-24T14:33:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5860000000000005e-06 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203421+0000", 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1056e-05 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203434+0000", 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3093e-05 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203623+0000", 2024-04-24T14:33:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000188646 2024-04-24T14:33:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203871+0000", 2024-04-24T14:33:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00024812199999999998 2024-04-24T14:33:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.27488403300000003, 2024-04-24T14:33:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059468100000000001, 2024-04-24T14:33:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203449+0000", 2024-04-24T14:33:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203469+0000", 2024-04-24T14:33:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0003000000000001e-05 2024-04-24T14:33:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203486+0000", 2024-04-24T14:33:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7309999999999999e-05 2024-04-24T14:33:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203501+0000", 2024-04-24T14:33:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.49e-05 2024-04-24T14:33:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203923+0000", 2024-04-24T14:33:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042195600000000002 2024-04-24T14:33:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204045+0000", 2024-04-24T14:33:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012202200000000001 2024-04-24T14:33:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:3 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.27351310499999998, 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00093837399999999996, 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 3 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204819+0000", 2024-04-24T14:33:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204833+0000", 2024-04-24T14:33:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3346999999999999e-05 2024-04-24T14:33:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204866+0000", 2024-04-24T14:33:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3302000000000001e-05 2024-04-24T14:33:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204903+0000", 2024-04-24T14:33:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7576000000000001e-05 2024-04-24T14:33:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205040+0000", 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013616 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205231+0000", 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019118999999999999 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205610+0000", 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037947000000000002 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205760+0000", 2024-04-24T14:33:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000149259 2024-04-24T14:33:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:13.493 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:13.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:13.495 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:13.499+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:13.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T14:33:13.595 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:13.599+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:13.606 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:13.606 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:13.606 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:13.606 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:13.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:13.696 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:13.698+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:13.723 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:13.723 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:13.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:13.797 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:13.799+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:13.835 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:13.835 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:13.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:13.836 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:13.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.836 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:36 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'22, mlcod=13'17)", 2024-04-24T14:33:13.836 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:13.836 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.507453126000001, 2024-04-24T14:33:13.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043791400000000001, 2024-04-24T14:33:13.836 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.836 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.836 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:13.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:13.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331851+0000", 2024-04-24T14:33:13.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331864+0000", 2024-04-24T14:33:13.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2767e-05 2024-04-24T14:33:13.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331881+0000", 2024-04-24T14:33:13.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6586e-05 2024-04-24T14:33:13.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331894+0000", 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2976000000000001e-05 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332161+0000", 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026765399999999998 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332197+0000", 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5565999999999998e-05 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336158+0000", 2024-04-24T14:33:13.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039616820000000002 2024-04-24T14:33:13.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336232+0000", 2024-04-24T14:33:13.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3151999999999994e-05 2024-04-24T14:33:13.840 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.840 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.840 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.840 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:37 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'23, mlcod=13'18)", 2024-04-24T14:33:13.840 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.506566648000003, 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040417389999999999, 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332738+0000", 2024-04-24T14:33:13.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:13.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332750+0000", 2024-04-24T14:33:13.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2432000000000001e-05 2024-04-24T14:33:13.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332768+0000", 2024-04-24T14:33:13.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7314000000000001e-05 2024-04-24T14:33:13.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332780+0000", 2024-04-24T14:33:13.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2716e-05 2024-04-24T14:33:13.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332925+0000", 2024-04-24T14:33:13.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144587 2024-04-24T14:33:13.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332955+0000", 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0246000000000001e-05 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336698+0000", 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037430559999999998 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336781+0000", 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2447999999999994e-05 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:39 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'25, mlcod=13'22)", 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.500448325000001, 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038736740000000001, 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:13.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338856+0000", 2024-04-24T14:33:13.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338995+0000", 2024-04-24T14:33:13.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013868100000000001 2024-04-24T14:33:13.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339029+0000", 2024-04-24T14:33:13.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4372000000000002e-05 2024-04-24T14:33:13.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339050+0000", 2024-04-24T14:33:13.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0656999999999999e-05 2024-04-24T14:33:13.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339156+0000", 2024-04-24T14:33:13.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106177 2024-04-24T14:33:13.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339188+0000", 2024-04-24T14:33:13.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.243e-05 2024-04-24T14:33:13.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342648+0000", 2024-04-24T14:33:13.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034601689999999999 2024-04-24T14:33:13.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342731+0000", 2024-04-24T14:33:13.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2534999999999995e-05 2024-04-24T14:33:13.849 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.849 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.849 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.849 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:67 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'41, mlcod=13'39)", 2024-04-24T14:33:13.849 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:13.850 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.449712040999998, 2024-04-24T14:33:13.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031029009999999999, 2024-04-24T14:33:13.850 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.850 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.850 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:13.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389592+0000", 2024-04-24T14:33:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389603+0000", 2024-04-24T14:33:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090000000000001e-05 2024-04-24T14:33:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389631+0000", 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7772000000000001e-05 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389646+0000", 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.432e-05 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389732+0000", 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6331999999999999e-05 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389771+0000", 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8751999999999998e-05 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392585+0000", 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028147469999999998 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392696+0000", 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111121 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.853 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.854 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.854 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T14:33:13.854 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:13.854 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.448569429999999, 2024-04-24T14:33:13.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030475519999999998, 2024-04-24T14:33:13.854 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.854 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.854 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390735+0000", 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390804+0000", 2024-04-24T14:33:13.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9488999999999998e-05 2024-04-24T14:33:13.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390832+0000", 2024-04-24T14:33:13.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7946e-05 2024-04-24T14:33:13.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390847+0000", 2024-04-24T14:33:13.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4693e-05 2024-04-24T14:33:13.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391508+0000", 2024-04-24T14:33:13.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066154799999999998 2024-04-24T14:33:13.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391538+0000", 2024-04-24T14:33:13.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0029e-05 2024-04-24T14:33:13.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393698+0000", 2024-04-24T14:33:13.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002160052 2024-04-24T14:33:13.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393784+0000", 2024-04-24T14:33:13.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5414999999999995e-05 2024-04-24T14:33:13.858 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.858 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.858 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.858 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:72 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'46, mlcod=13'43)", 2024-04-24T14:33:13.858 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:13.858 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.442395937000001, 2024-04-24T14:33:13.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036059460000000001, 2024-04-24T14:33:13.859 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.859 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.859 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:13.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:13.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396908+0000", 2024-04-24T14:33:13.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396968+0000", 2024-04-24T14:33:13.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9412e-05 2024-04-24T14:33:13.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397010+0000", 2024-04-24T14:33:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2169000000000002e-05 2024-04-24T14:33:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397022+0000", 2024-04-24T14:33:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1749999999999999e-05 2024-04-24T14:33:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397096+0000", 2024-04-24T14:33:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4701000000000002e-05 2024-04-24T14:33:13.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397126+0000", 2024-04-24T14:33:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2024-04-24T14:33:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400443+0000", 2024-04-24T14:33:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033172140000000002 2024-04-24T14:33:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400516+0000", 2024-04-24T14:33:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2181999999999994e-05 2024-04-24T14:33:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:73 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'47, mlcod=13'43)", 2024-04-24T14:33:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.441548533000002, 2024-04-24T14:33:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055152170000000002, 2024-04-24T14:33:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.863 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.864 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:13.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:13.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397756+0000", 2024-04-24T14:33:13.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397884+0000", 2024-04-24T14:33:13.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012842100000000001 2024-04-24T14:33:13.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397923+0000", 2024-04-24T14:33:13.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8942000000000001e-05 2024-04-24T14:33:13.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397939+0000", 2024-04-24T14:33:13.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6073e-05 2024-04-24T14:33:13.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398106+0000", 2024-04-24T14:33:13.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016737100000000001 2024-04-24T14:33:13.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398137+0000", 2024-04-24T14:33:13.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0496000000000001e-05 2024-04-24T14:33:13.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403191+0000", 2024-04-24T14:33:13.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00505391 2024-04-24T14:33:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403272+0000", 2024-04-24T14:33:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1483999999999994e-05 2024-04-24T14:33:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:74 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'48, mlcod=13'43)", 2024-04-24T14:33:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.440749507, 2024-04-24T14:33:13.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048091760000000001, 2024-04-24T14:33:13.869 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.869 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.869 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:13.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:13.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398555+0000", 2024-04-24T14:33:13.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398570+0000", 2024-04-24T14:33:13.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.558e-05 2024-04-24T14:33:13.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398610+0000", 2024-04-24T14:33:13.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0028999999999999e-05 2024-04-24T14:33:13.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398625+0000", 2024-04-24T14:33:13.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-04-24T14:33:13.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398701+0000", 2024-04-24T14:33:13.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6507000000000005e-05 2024-04-24T14:33:13.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398731+0000", 2024-04-24T14:33:13.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9356000000000002e-05 2024-04-24T14:33:13.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403281+0000", 2024-04-24T14:33:13.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004549796 2024-04-24T14:33:13.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403365+0000", 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4685000000000007e-05 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.419436582000003, 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038140460000000002, 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:13.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:13.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419868+0000", 2024-04-24T14:33:13.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419882+0000", 2024-04-24T14:33:13.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.45e-05 2024-04-24T14:33:13.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419919+0000", 2024-04-24T14:33:13.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6939e-05 2024-04-24T14:33:13.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419961+0000", 2024-04-24T14:33:13.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2148999999999999e-05 2024-04-24T14:33:13.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420084+0000", 2024-04-24T14:33:13.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122958 2024-04-24T14:33:13.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420122+0000", 2024-04-24T14:33:13.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8334999999999998e-05 2024-04-24T14:33:13.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423616+0000", 2024-04-24T14:33:13.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003493566 2024-04-24T14:33:13.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423683+0000", 2024-04-24T14:33:13.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7149000000000004e-05 2024-04-24T14:33:13.877 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.877 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.877 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.877 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T14:33:13.877 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:13.878 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.418024826999996, 2024-04-24T14:33:13.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030290730000000002, 2024-04-24T14:33:13.878 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.878 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.878 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:13.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421279+0000", 2024-04-24T14:33:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421352+0000", 2024-04-24T14:33:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2251999999999999e-05 2024-04-24T14:33:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421378+0000", 2024-04-24T14:33:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6006000000000001e-05 2024-04-24T14:33:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421388+0000", 2024-04-24T14:33:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0373e-05 2024-04-24T14:33:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421473+0000", 2024-04-24T14:33:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4665000000000003e-05 2024-04-24T14:33:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421512+0000", 2024-04-24T14:33:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9619000000000001e-05 2024-04-24T14:33:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424198+0000", 2024-04-24T14:33:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026854610000000001 2024-04-24T14:33:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424310+0000", 2024-04-24T14:33:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011208 2024-04-24T14:33:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:87 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'57, mlcod=13'51)", 2024-04-24T14:33:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.413761868000002, 2024-04-24T14:33:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002974057, 2024-04-24T14:33:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425542+0000", 2024-04-24T14:33:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425614+0000", 2024-04-24T14:33:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1328999999999994e-05 2024-04-24T14:33:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425664+0000", 2024-04-24T14:33:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0491999999999997e-05 2024-04-24T14:33:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425676+0000", 2024-04-24T14:33:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.134e-05 2024-04-24T14:33:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425798+0000", 2024-04-24T14:33:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012221700000000001 2024-04-24T14:33:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425860+0000", 2024-04-24T14:33:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1761999999999996e-05 2024-04-24T14:33:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428368+0000", 2024-04-24T14:33:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025084 2024-04-24T14:33:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428518+0000", 2024-04-24T14:33:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149884 2024-04-24T14:33:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.213653143000002, 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002919694, 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625651+0000", 2024-04-24T14:33:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625667+0000", 2024-04-24T14:33:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5973000000000001e-05 2024-04-24T14:33:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625689+0000", 2024-04-24T14:33:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:33:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625705+0000", 2024-04-24T14:33:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5797000000000001e-05 2024-04-24T14:33:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625805+0000", 2024-04-24T14:33:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9238000000000001e-05 2024-04-24T14:33:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625837+0000", 2024-04-24T14:33:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2722000000000003e-05 2024-04-24T14:33:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628476+0000", 2024-04-24T14:33:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026391980000000002 2024-04-24T14:33:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628572+0000", 2024-04-24T14:33:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5754000000000005e-05 2024-04-24T14:33:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'88)", 2024-04-24T14:33:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.773715265, 2024-04-24T14:33:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031994179999999999, 2024-04-24T14:33:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065589+0000", 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065602+0000", 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2737e-05 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065620+0000", 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8546000000000002e-05 2024-04-24T14:33:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065629+0000", 2024-04-24T14:33:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1600000000000004e-06 2024-04-24T14:33:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065729+0000", 2024-04-24T14:33:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9933999999999996e-05 2024-04-24T14:33:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065792+0000", 2024-04-24T14:33:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2712000000000005e-05 2024-04-24T14:33:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068667+0000", 2024-04-24T14:33:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028752920000000002 2024-04-24T14:33:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068790+0000", 2024-04-24T14:33:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012244400000000001 2024-04-24T14:33:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:137 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'91)", 2024-04-24T14:33:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.772359528999999, 2024-04-24T14:33:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052524399999999997, 2024-04-24T14:33:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:13.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066945+0000", 2024-04-24T14:33:13.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067031+0000", 2024-04-24T14:33:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6228000000000002e-05 2024-04-24T14:33:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067060+0000", 2024-04-24T14:33:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8445999999999998e-05 2024-04-24T14:33:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067069+0000", 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999996e-06 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067172+0000", 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010285099999999999 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067234+0000", 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2068000000000003e-05 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072090+0000", 2024-04-24T14:33:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048560859999999999 2024-04-24T14:33:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072199+0000", 2024-04-24T14:33:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108384 2024-04-24T14:33:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:138 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'96, mlcod=13'91)", 2024-04-24T14:33:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.771359765, 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053159380000000001, 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067945+0000", 2024-04-24T14:33:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067957+0000", 2024-04-24T14:33:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1963e-05 2024-04-24T14:33:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067982+0000", 2024-04-24T14:33:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5570000000000001e-05 2024-04-24T14:33:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067994+0000", 2024-04-24T14:33:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2306e-05 2024-04-24T14:33:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068282+0000", 2024-04-24T14:33:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028799000000000001 2024-04-24T14:33:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068307+0000", 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4351000000000001e-05 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073142+0000", 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004835326 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073262+0000", 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119695 2024-04-24T14:33:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:139 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.769876938000003, 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039447479999999997, 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069428+0000", 2024-04-24T14:33:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069440+0000", 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2047e-05 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069469+0000", 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8852000000000001e-05 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069477+0000", 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069531+0000", 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4126999999999999e-05 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069559+0000", 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7881e-05 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073270+0000", 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037104519999999999 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073373+0000", 2024-04-24T14:33:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010380899999999999 2024-04-24T14:33:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:140 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'92)", 2024-04-24T14:33:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.769051062000003, 2024-04-24T14:33:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031722510000000001, 2024-04-24T14:33:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070253+0000", 2024-04-24T14:33:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070307+0000", 2024-04-24T14:33:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3909000000000002e-05 2024-04-24T14:33:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070330+0000", 2024-04-24T14:33:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2599000000000002e-05 2024-04-24T14:33:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070337+0000", 2024-04-24T14:33:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3030000000000002e-06 2024-04-24T14:33:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070377+0000", 2024-04-24T14:33:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9916000000000002e-05 2024-04-24T14:33:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070398+0000", 2024-04-24T14:33:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0579000000000001e-05 2024-04-24T14:33:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073377+0000", 2024-04-24T14:33:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002979013 2024-04-24T14:33:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073427+0000", 2024-04-24T14:33:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0078999999999999e-05 2024-04-24T14:33:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:143 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'101, mlcod=13'98)", 2024-04-24T14:33:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.762865781999999, 2024-04-24T14:33:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030247590000000001, 2024-04-24T14:33:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076438+0000", 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076533+0000", 2024-04-24T14:33:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4443000000000004e-05 2024-04-24T14:33:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076563+0000", 2024-04-24T14:33:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0386999999999999e-05 2024-04-24T14:33:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076604+0000", 2024-04-24T14:33:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1012e-05 2024-04-24T14:33:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076711+0000", 2024-04-24T14:33:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010637300000000001 2024-04-24T14:33:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076737+0000", 2024-04-24T14:33:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6190000000000002e-05 2024-04-24T14:33:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079368+0000", 2024-04-24T14:33:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026313529999999999 2024-04-24T14:33:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079465+0000", 2024-04-24T14:33:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6426000000000001e-05 2024-04-24T14:33:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:159 1.0 e14/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'113, mlcod=14'109)", 2024-04-24T14:33:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.129045675999997, 2024-04-24T14:33:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052053589999999997, 2024-04-24T14:33:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710259+0000", 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710272+0000", 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.382e-05 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710303+0000", 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0629000000000001e-05 2024-04-24T14:33:13.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710322+0000", 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8859999999999999e-05 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710479+0000", 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157076 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710535+0000", 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6289000000000003e-05 2024-04-24T14:33:13.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715391+0000", 2024-04-24T14:33:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048557139999999997 2024-04-24T14:33:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715465+0000", 2024-04-24T14:33:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4301000000000006e-05 2024-04-24T14:33:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.922 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.923 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:161 1.0 e14/12 1:30306672:devicehealth::main.db.0000000000000000:head v 14'115, mlcod=14'112)", 2024-04-24T14:33:13.923 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:13.923 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.121934873999997, 2024-04-24T14:33:13.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039960450000000002, 2024-04-24T14:33:13.923 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:13.923 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:13.923 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:13.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717370+0000", 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:13.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717499+0000", 2024-04-24T14:33:13.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012877400000000001 2024-04-24T14:33:13.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:13.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717513+0000", 2024-04-24T14:33:13.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4229999999999999e-05 2024-04-24T14:33:13.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:13.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717521+0000", 2024-04-24T14:33:13.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2024-04-24T14:33:13.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:13.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718040+0000", 2024-04-24T14:33:13.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051920099999999995 2024-04-24T14:33:13.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:13.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718094+0000", 2024-04-24T14:33:13.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859000000000001e-05 2024-04-24T14:33:13.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:13.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721278+0000", 2024-04-24T14:33:13.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031838560000000001 2024-04-24T14:33:13.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:13.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:13.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:13.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721367+0000", 2024-04-24T14:33:13.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9241999999999997e-05 2024-04-24T14:33:13.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.927 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:13.927 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:13.927 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:13.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:13.929 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:13.934+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:14.029 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:14.034+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:14.031 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:14.031 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:14.031 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:14.031 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:14.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:14.130 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:14.135+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:14.136 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:14.136 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:14.136 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:14.136 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:14.136 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:14.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:14.231 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:14.234+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:14.242 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:14.242 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:14.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:14.242 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:14.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.242 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:1 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:14.243 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:14.243 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.0422521469999999, 2024-04-24T14:33:14.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038534000000000002, 2024-04-24T14:33:14.243 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:14.243 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:14.243 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:14.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:14.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:14.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:14.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:14.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:14.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:14.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:14.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206414+0000", 2024-04-24T14:33:14.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2024-04-24T14:33:14.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:14.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206427+0000", 2024-04-24T14:33:14.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3457e-05 2024-04-24T14:33:14.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206450+0000", 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2966999999999999e-05 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206464+0000", 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3706999999999999e-05 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206581+0000", 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117161 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:14.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206802+0000", 2024-04-24T14:33:14.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221108 2024-04-24T14:33:14.246 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:14.246 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:14.246 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:14.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.246 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:14.246 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:14.246 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.042175863, 2024-04-24T14:33:14.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056574800000000003, 2024-04-24T14:33:14.246 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:14.247 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:14.247 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:14.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:14.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:14.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:14.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:14.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:14.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:14.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:14.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206492+0000", 2024-04-24T14:33:14.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:14.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:14.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206497+0000", 2024-04-24T14:33:14.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6570000000000002e-06 2024-04-24T14:33:14.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:14.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206524+0000", 2024-04-24T14:33:14.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6503e-05 2024-04-24T14:33:14.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206535+0000", 2024-04-24T14:33:14.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.092e-05 2024-04-24T14:33:14.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:14.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206859+0000", 2024-04-24T14:33:14.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032470600000000002 2024-04-24T14:33:14.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:14.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207059+0000", 2024-04-24T14:33:14.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000199132 2024-04-24T14:33:14.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:14.250 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:14.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:14.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.250 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:3 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:14.250 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:14.250 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.0420736049999999, 2024-04-24T14:33:14.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067768899999999998, 2024-04-24T14:33:14.250 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:14.250 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:14.250 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:14.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:14.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:14.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:14.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:14.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:14.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:14.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:14.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206594+0000", 2024-04-24T14:33:14.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:14.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:14.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206601+0000", 2024-04-24T14:33:14.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7939999999999997e-06 2024-04-24T14:33:14.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:14.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206611+0000", 2024-04-24T14:33:14.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.008e-05 2024-04-24T14:33:14.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206617+0000", 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4629999999999998e-06 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207107+0000", 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048966400000000001 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207273+0000", 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016570199999999999 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:14.253 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:14.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:14.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:14.254 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:14.254 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:14.255 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:14.332 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:14.334+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:14.364 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:14.364 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:14.364 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:14.364 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:14.374 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:14.433 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:14.434+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:14.471 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:14.471 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:14.471 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:14.471 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:14.471 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:14.480 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:14.534 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:14.534+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:14.566 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-24T14:33:14.585 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:14.585 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:14.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:14.585 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:14.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.585 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.585 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:14.585 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 52.262223249000002, 2024-04-24T14:33:14.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011830553000000001, 2024-04-24T14:33:14.585 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.585 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.586 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.586 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.586 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.586 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 36 2024-04-24T14:33:14.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.586 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:14.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326000+0000", 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326006+0000", 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9340000000000003e-06 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326044+0000", 2024-04-24T14:33:14.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8124999999999998e-05 2024-04-24T14:33:14.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326054+0000", 2024-04-24T14:33:14.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9650000000000003e-06 2024-04-24T14:33:14.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332746+0000", 2024-04-24T14:33:14.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006692539 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332833+0000", 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6482000000000003e-05 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332947+0000", 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114472 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334320+0000", 2024-04-24T14:33:14.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001372489 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337717+0000", 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003396849 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337789+0000", 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2080000000000001e-05 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337831+0000", 2024-04-24T14:33:14.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.244e-05 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 52.262161386999999, 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.591 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:33:14.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:33:14.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:33:14.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:33:14.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:33:14.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:33:14.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:33:14.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:33:14.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:33:14.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:33:14.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:33:14.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:33:14.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.596 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 52.168656914000003, 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011218514000000001, 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 88 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:14.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419565+0000", 2024-04-24T14:33:14.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:14.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419575+0000", 2024-04-24T14:33:14.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5340000000000005e-06 2024-04-24T14:33:14.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419616+0000", 2024-04-24T14:33:14.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1301999999999999e-05 2024-04-24T14:33:14.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419638+0000", 2024-04-24T14:33:14.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2177000000000001e-05 2024-04-24T14:33:14.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428578+0000", 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089398280000000004 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428665+0000", 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6644000000000006e-05 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428797+0000", 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013231500000000001 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.429866+0000", 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001069221 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430684+0000", 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081788399999999995 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430743+0000", 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8875999999999997e-05 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430786+0000", 2024-04-24T14:33:14.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2398999999999998e-05 2024-04-24T14:33:14.602 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.602 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.602 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.602 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.602 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:14.602 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.962646444999997, 2024-04-24T14:33:14.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:33:14.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:33:14.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:33:14.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:33:14.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:33:14.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:33:14.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:33:14.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.607 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.608 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:14.608 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.962420676999997, 2024-04-24T14:33:14.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:33:14.608 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.608 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.608 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.608 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.608 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.608 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:33:14.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.608 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:14.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:14.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:33:14.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:14.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:33:14.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:33:14.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:33:14.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:33:14.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:33:14.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:33:14.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:33:14.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:33:14.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:33:14.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:33:14.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:33:14.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:14.613 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.96224359, 2024-04-24T14:33:14.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:33:14.614 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.614 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.614 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.614 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.614 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.614 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:33:14.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.614 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:14.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:33:14.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:33:14.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:33:14.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:33:14.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:33:14.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:33:14.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:33:14.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:33:14.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:33:14.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:33:14.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:33:14.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:33:14.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:33:14.619 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.619 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.619 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.619 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.619 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:14.619 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.962139676, 2024-04-24T14:33:14.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:33:14.620 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.620 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.620 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.620 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.620 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.620 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:33:14.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.620 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:14.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:14.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:33:14.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:14.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:33:14.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:33:14.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:33:14.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:33:14.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:33:14.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:33:14.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:33:14.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:33:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:33:14.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:33:14.625 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.625 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.625 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.625 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.625 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:14.625 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.961843356000003, 2024-04-24T14:33:14.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:33:14.625 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.625 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.625 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.626 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:33:14.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.626 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:14.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:14.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:33:14.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:14.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:33:14.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:33:14.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:33:14.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:33:14.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:33:14.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:33:14.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:33:14.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:33:14.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:33:14.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:33:14.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:33:14.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:33:14.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:33:14.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:33:14.631 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.631 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.631 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.631 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.631 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:14.631 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.961751225, 2024-04-24T14:33:14.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:33:14.631 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.632 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.632 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.632 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.632 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.632 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:33:14.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.632 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:14.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:33:14.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:33:14.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:33:14.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:33:14.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:33:14.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:33:14.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:33:14.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:33:14.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:33:14.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:33:14.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:33:14.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.961666012999999, 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.637 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.638 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.638 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.638 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:33:14.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.638 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:14.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:14.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:33:14.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:14.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:33:14.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:33:14.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:33:14.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:33:14.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:33:14.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:33:14.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:33:14.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:33:14.642 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.642 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.642 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.961429131999999, 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:14.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:14.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:33:14.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:33:14.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:33:14.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:33:14.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:33:14.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:33:14.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:33:14.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:33:14.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:33:14.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:33:14.648 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.648 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.648 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.648 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.649 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:14.649 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.961347609999997, 2024-04-24T14:33:14.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:33:14.649 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.649 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.649 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.649 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.649 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.649 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:33:14.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.649 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:33:14.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:33:14.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:33:14.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:33:14.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:33:14.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:33:14.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:33:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:33:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:33:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:33:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:33:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:33:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:33:14.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:33:14.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:33:14.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:33:14.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.529628533999997, 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01155111, 2024-04-24T14:33:14.654 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058602+0000", 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1239999999999999e-06 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058609+0000", 2024-04-24T14:33:14.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6719999999999998e-06 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058626+0000", 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7569999999999999e-05 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065271+0000", 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00664469 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065398+0000", 2024-04-24T14:33:14.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012694299999999999 2024-04-24T14:33:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065598+0000", 2024-04-24T14:33:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000200216 2024-04-24T14:33:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067596+0000", 2024-04-24T14:33:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019976450000000001 2024-04-24T14:33:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070055+0000", 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024593710000000001 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070109+0000", 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3690999999999998e-05 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070147+0000", 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7910000000000001e-05 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.659 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.529589199, 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:33:14.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:33:14.665 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.665 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.665 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.665 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.665 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:14.665 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.529547696000002, 2024-04-24T14:33:14.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:33:14.665 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.666 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.666 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.666 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.666 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.666 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:33:14.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.666 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:14.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:14.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:14.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:14.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:33:14.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:33:14.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:33:14.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:33:14.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:33:14.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:33:14.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:33:14.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:33:14.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:33:14.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:33:14.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:33:14.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:33:14.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:33:14.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:33:14.671 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.671 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.671 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.671 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.671 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:14.671 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.529435263000003, 2024-04-24T14:33:14.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:33:14.671 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:14.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:33:14.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:14.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:33:14.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:33:14.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:33:14.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:33:14.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:33:14.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:33:14.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:33:14.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:33:14.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:33:14.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:33:14.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:33:14.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:33:14.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:33:14.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:33:14.676 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.677 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.677 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.677 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.677 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:14.677 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.529406963, 2024-04-24T14:33:14.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:33:14.677 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.677 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.677 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.678 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.678 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.678 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:33:14.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.678 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:14.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:14.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:33:14.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:14.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:33:14.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:33:14.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:33:14.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:33:14.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:33:14.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:33:14.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:33:14.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:33:14.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:33:14.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:33:14.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:33:14.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:33:14.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:33:14.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:33:14.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:33:14.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:33:14.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:33:14.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:33:14.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:33:14.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:33:14.682 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.683 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.683 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.683 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.683 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:14.683 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.529319258999998, 2024-04-24T14:33:14.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:33:14.683 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.683 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.683 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.683 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.683 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.683 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:33:14.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.684 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:14.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:14.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:33:14.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:14.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:33:14.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:33:14.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:33:14.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:33:14.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:33:14.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:33:14.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:33:14.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:33:14.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:33:14.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:33:14.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:33:14.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:33:14.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:33:14.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:33:14.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:33:14.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:33:14.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:33:14.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:33:14.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:33:14.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:33:14.688 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.689 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.689 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.689 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:14.689 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:14.689 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.529196458000001, 2024-04-24T14:33:14.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:33:14.689 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.689 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.689 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.689 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.690 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.690 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:33:14.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.690 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:14.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:14.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:33:14.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:14.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:33:14.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:33:14.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:33:14.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:33:14.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:33:14.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:33:14.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:33:14.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:33:14.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:33:14.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:33:14.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:33:14.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:33:14.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:33:14.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:33:14.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:33:14.694 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.694 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.885956526000001, 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:14.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:14.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:14.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:14.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:14.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:14.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:14.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:33:14.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:14.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:33:14.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.698 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:14.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:33:14.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:33:14.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.698 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:14.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:33:14.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:33:14.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.698 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:14.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:14.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:33:14.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:33:14.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:14.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:14.700 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:14.700 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:33:14.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:33:14.700 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.700 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.700 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.700 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:14.700 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:14.700 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:14.702 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:14.702 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:14.702+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:14.803 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:14.807+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:14.851 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:14.851 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:14.851 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:14.851 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:14.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:33:14.860 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:14.904 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:14.906+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:14.963 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:14.964 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:14.964 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:14.964 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:14.964 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:14.972 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:14.999 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:15.004 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:15.006+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:15.022 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:15.044 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:15.064 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:15.076 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:15.076 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:15.084 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:15.085 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:15.085 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:15.085 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:15.085 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:15.085 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:15.085 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:15.086 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:15.088 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:15.131 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:15.134+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:15.206 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:15.206+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:15.233 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:15.233 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:15.242 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:15.307 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:15.310+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:15.339 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:15.340 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:15.348 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:15.408 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:15.411+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:15.447 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:15.447 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:15.456 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:15.509 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:15.511+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:15.565 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:15.565 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:15.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:15.618 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:15.623+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:15.710 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:15.710+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:15.722 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:15.722 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:15.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:15.811 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:15.814+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:15.833 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:15.833 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:15.843 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:15.912 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:15.914+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:15.944 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:15.944 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:15.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:16.013 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:16.015+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:16.053 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:16.053 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:16.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:16.114 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:16.114+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:16.161 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:16.161 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:16.171 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:16.215 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:16.218+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:16.281 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:16.281 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:16.291 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:16.316 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:16.318+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:16.393 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:16.394 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:16.403 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:16.447 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:16.450+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:16.518 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:16.519+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:16.547 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:16.547 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:16.556 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:16.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T14:33:16.619 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:16.622+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:16.655 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:16.655 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:16.664 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:16.720 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:16.722+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:16.763 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:16.763 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:16.772 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:16.820 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:16.822+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:16.919 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:16.919 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:16.921 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:16.923+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:16.928 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:17.021 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:17.022+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:17.028 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:17.029 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:17.038 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:17.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:33:17.123 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:17.126+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:17.183 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:17.184 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:17.193 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:17.224 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:17.227+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:17.228 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:17.248 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:17.269 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:17.291 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:17.293 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:17.293 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:17.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:17.313 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:17.314 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:17.314 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:17.314 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:17.314 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:17.314 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:17.315 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:17.319 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:17.362 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:17.367+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:17.405 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:17.405 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:17.405 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:17.405 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:17.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:17.425 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:17.427+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:17.517 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:17.517 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:17.518 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:17.518 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:17.518 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:17.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:17.527 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:17.526+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:17.582 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-24T14:33:17.627 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:17.627 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:17.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:17.627 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:17.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.628 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:36 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'22, mlcod=13'17)", 2024-04-24T14:33:17.628 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:17.628 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.299401951, 2024-04-24T14:33:17.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043791400000000001, 2024-04-24T14:33:17.628 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.628 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.628 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:17.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:17.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331851+0000", 2024-04-24T14:33:17.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331864+0000", 2024-04-24T14:33:17.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2767e-05 2024-04-24T14:33:17.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331881+0000", 2024-04-24T14:33:17.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6586e-05 2024-04-24T14:33:17.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331894+0000", 2024-04-24T14:33:17.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2976000000000001e-05 2024-04-24T14:33:17.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332161+0000", 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026765399999999998 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332197+0000", 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5565999999999998e-05 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336158+0000", 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039616820000000002 2024-04-24T14:33:17.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336232+0000", 2024-04-24T14:33:17.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3151999999999994e-05 2024-04-24T14:33:17.632 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.632 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.632 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.632 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:37 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'23, mlcod=13'18)", 2024-04-24T14:33:17.632 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:17.632 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.298515473000002, 2024-04-24T14:33:17.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040417389999999999, 2024-04-24T14:33:17.633 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.633 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.633 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:17.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:17.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332738+0000", 2024-04-24T14:33:17.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:17.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332750+0000", 2024-04-24T14:33:17.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2432000000000001e-05 2024-04-24T14:33:17.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332768+0000", 2024-04-24T14:33:17.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7314000000000001e-05 2024-04-24T14:33:17.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332780+0000", 2024-04-24T14:33:17.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2716e-05 2024-04-24T14:33:17.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332925+0000", 2024-04-24T14:33:17.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144587 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332955+0000", 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0246000000000001e-05 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336698+0000", 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037430559999999998 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336781+0000", 2024-04-24T14:33:17.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2447999999999994e-05 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:39 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'25, mlcod=13'22)", 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.292397149999999, 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038736740000000001, 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:17.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:17.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338856+0000", 2024-04-24T14:33:17.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338995+0000", 2024-04-24T14:33:17.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013868100000000001 2024-04-24T14:33:17.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339029+0000", 2024-04-24T14:33:17.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4372000000000002e-05 2024-04-24T14:33:17.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339050+0000", 2024-04-24T14:33:17.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0656999999999999e-05 2024-04-24T14:33:17.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339156+0000", 2024-04-24T14:33:17.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106177 2024-04-24T14:33:17.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339188+0000", 2024-04-24T14:33:17.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.243e-05 2024-04-24T14:33:17.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342648+0000", 2024-04-24T14:33:17.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034601689999999999 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342731+0000", 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2534999999999995e-05 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:67 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'41, mlcod=13'39)", 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.241660865999997, 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031029009999999999, 2024-04-24T14:33:17.641 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.642 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.642 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:17.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:17.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389592+0000", 2024-04-24T14:33:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389603+0000", 2024-04-24T14:33:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090000000000001e-05 2024-04-24T14:33:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389631+0000", 2024-04-24T14:33:17.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7772000000000001e-05 2024-04-24T14:33:17.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389646+0000", 2024-04-24T14:33:17.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.432e-05 2024-04-24T14:33:17.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389732+0000", 2024-04-24T14:33:17.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6331999999999999e-05 2024-04-24T14:33:17.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389771+0000", 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8751999999999998e-05 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392585+0000", 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028147469999999998 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392696+0000", 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111121 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.645 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.240518254999998, 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030475519999999998, 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390735+0000", 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390804+0000", 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9488999999999998e-05 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390832+0000", 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7946e-05 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390847+0000", 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4693e-05 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391508+0000", 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066154799999999998 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391538+0000", 2024-04-24T14:33:17.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0029e-05 2024-04-24T14:33:17.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393698+0000", 2024-04-24T14:33:17.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002160052 2024-04-24T14:33:17.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393784+0000", 2024-04-24T14:33:17.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5414999999999995e-05 2024-04-24T14:33:17.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.650 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.650 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:72 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'46, mlcod=13'43)", 2024-04-24T14:33:17.650 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:17.650 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.234344761999999, 2024-04-24T14:33:17.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036059460000000001, 2024-04-24T14:33:17.650 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.650 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.650 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:17.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:17.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396908+0000", 2024-04-24T14:33:17.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396968+0000", 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9412e-05 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397010+0000", 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2169000000000002e-05 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397022+0000", 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1749999999999999e-05 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397096+0000", 2024-04-24T14:33:17.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4701000000000002e-05 2024-04-24T14:33:17.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397126+0000", 2024-04-24T14:33:17.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2024-04-24T14:33:17.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400443+0000", 2024-04-24T14:33:17.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033172140000000002 2024-04-24T14:33:17.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400516+0000", 2024-04-24T14:33:17.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2181999999999994e-05 2024-04-24T14:33:17.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.654 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.654 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:73 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'47, mlcod=13'43)", 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.233497358000001, 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055152170000000002, 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397756+0000", 2024-04-24T14:33:17.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397884+0000", 2024-04-24T14:33:17.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012842100000000001 2024-04-24T14:33:17.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397923+0000", 2024-04-24T14:33:17.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8942000000000001e-05 2024-04-24T14:33:17.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397939+0000", 2024-04-24T14:33:17.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6073e-05 2024-04-24T14:33:17.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398106+0000", 2024-04-24T14:33:17.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016737100000000001 2024-04-24T14:33:17.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398137+0000", 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0496000000000001e-05 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403191+0000", 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00505391 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403272+0000", 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1483999999999994e-05 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.658 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:74 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'48, mlcod=13'43)", 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.232698331999998, 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048091760000000001, 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398555+0000", 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398570+0000", 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.558e-05 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398610+0000", 2024-04-24T14:33:17.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0028999999999999e-05 2024-04-24T14:33:17.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398625+0000", 2024-04-24T14:33:17.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-04-24T14:33:17.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398701+0000", 2024-04-24T14:33:17.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6507000000000005e-05 2024-04-24T14:33:17.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398731+0000", 2024-04-24T14:33:17.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9356000000000002e-05 2024-04-24T14:33:17.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403281+0000", 2024-04-24T14:33:17.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004549796 2024-04-24T14:33:17.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403365+0000", 2024-04-24T14:33:17.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4685000000000007e-05 2024-04-24T14:33:17.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.663 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.663 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-04-24T14:33:17.663 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:17.663 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.211385407000002, 2024-04-24T14:33:17.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038140460000000002, 2024-04-24T14:33:17.663 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.663 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.663 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419868+0000", 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419882+0000", 2024-04-24T14:33:17.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.45e-05 2024-04-24T14:33:17.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419919+0000", 2024-04-24T14:33:17.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6939e-05 2024-04-24T14:33:17.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419961+0000", 2024-04-24T14:33:17.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2148999999999999e-05 2024-04-24T14:33:17.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420084+0000", 2024-04-24T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122958 2024-04-24T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420122+0000", 2024-04-24T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8334999999999998e-05 2024-04-24T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423616+0000", 2024-04-24T14:33:17.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003493566 2024-04-24T14:33:17.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423683+0000", 2024-04-24T14:33:17.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7149000000000004e-05 2024-04-24T14:33:17.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.667 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.667 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T14:33:17.667 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:17.667 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.209973652000002, 2024-04-24T14:33:17.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030290730000000002, 2024-04-24T14:33:17.668 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.668 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.668 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:17.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:17.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421279+0000", 2024-04-24T14:33:17.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421352+0000", 2024-04-24T14:33:17.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2251999999999999e-05 2024-04-24T14:33:17.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421378+0000", 2024-04-24T14:33:17.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6006000000000001e-05 2024-04-24T14:33:17.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421388+0000", 2024-04-24T14:33:17.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0373e-05 2024-04-24T14:33:17.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421473+0000", 2024-04-24T14:33:17.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4665000000000003e-05 2024-04-24T14:33:17.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421512+0000", 2024-04-24T14:33:17.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9619000000000001e-05 2024-04-24T14:33:17.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424198+0000", 2024-04-24T14:33:17.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026854610000000001 2024-04-24T14:33:17.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424310+0000", 2024-04-24T14:33:17.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011208 2024-04-24T14:33:17.671 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.671 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:87 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'57, mlcod=13'51)", 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.205710693, 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002974057, 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:17.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425542+0000", 2024-04-24T14:33:17.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425614+0000", 2024-04-24T14:33:17.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1328999999999994e-05 2024-04-24T14:33:17.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425664+0000", 2024-04-24T14:33:17.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0491999999999997e-05 2024-04-24T14:33:17.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425676+0000", 2024-04-24T14:33:17.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.134e-05 2024-04-24T14:33:17.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425798+0000", 2024-04-24T14:33:17.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012221700000000001 2024-04-24T14:33:17.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425860+0000", 2024-04-24T14:33:17.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1761999999999996e-05 2024-04-24T14:33:17.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428368+0000", 2024-04-24T14:33:17.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025084 2024-04-24T14:33:17.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428518+0000", 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149884 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.005601968000001, 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002919694, 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.676 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:17.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:17.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625651+0000", 2024-04-24T14:33:17.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:17.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625667+0000", 2024-04-24T14:33:17.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5973000000000001e-05 2024-04-24T14:33:17.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625689+0000", 2024-04-24T14:33:17.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:33:17.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625705+0000", 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5797000000000001e-05 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625805+0000", 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9238000000000001e-05 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625837+0000", 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2722000000000003e-05 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628476+0000", 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026391980000000002 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628572+0000", 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5754000000000005e-05 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'88)", 2024-04-24T14:33:17.680 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:17.681 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.565664089999999, 2024-04-24T14:33:17.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031994179999999999, 2024-04-24T14:33:17.681 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.681 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.681 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:17.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:17.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065589+0000", 2024-04-24T14:33:17.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065602+0000", 2024-04-24T14:33:17.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2737e-05 2024-04-24T14:33:17.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065620+0000", 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8546000000000002e-05 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065629+0000", 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1600000000000004e-06 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065729+0000", 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9933999999999996e-05 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065792+0000", 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2712000000000005e-05 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068667+0000", 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028752920000000002 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068790+0000", 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012244400000000001 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.684 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:137 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'91)", 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.564308353999998, 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052524399999999997, 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:17.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066945+0000", 2024-04-24T14:33:17.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067031+0000", 2024-04-24T14:33:17.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6228000000000002e-05 2024-04-24T14:33:17.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067060+0000", 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8445999999999998e-05 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067069+0000", 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999996e-06 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067172+0000", 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010285099999999999 2024-04-24T14:33:17.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067234+0000", 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2068000000000003e-05 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072090+0000", 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048560859999999999 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072199+0000", 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108384 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.688 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.689 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.689 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:138 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'96, mlcod=13'91)", 2024-04-24T14:33:17.689 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:17.689 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.563308589999998, 2024-04-24T14:33:17.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053159380000000001, 2024-04-24T14:33:17.689 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.689 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.689 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:17.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067945+0000", 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067957+0000", 2024-04-24T14:33:17.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1963e-05 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067982+0000", 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5570000000000001e-05 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067994+0000", 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2306e-05 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068282+0000", 2024-04-24T14:33:17.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028799000000000001 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068307+0000", 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4351000000000001e-05 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073142+0000", 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004835326 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073262+0000", 2024-04-24T14:33:17.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119695 2024-04-24T14:33:17.693 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.693 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.693 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.693 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:139 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T14:33:17.693 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:17.693 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.561825763000002, 2024-04-24T14:33:17.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039447479999999997, 2024-04-24T14:33:17.693 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.693 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.693 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:17.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:17.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069428+0000", 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069440+0000", 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2047e-05 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069469+0000", 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8852000000000001e-05 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069477+0000", 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069531+0000", 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4126999999999999e-05 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069559+0000", 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7881e-05 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073270+0000", 2024-04-24T14:33:17.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037104519999999999 2024-04-24T14:33:17.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073373+0000", 2024-04-24T14:33:17.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010380899999999999 2024-04-24T14:33:17.697 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.697 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.697 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.697 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:140 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'92)", 2024-04-24T14:33:17.697 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:17.698 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.560999887000001, 2024-04-24T14:33:17.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031722510000000001, 2024-04-24T14:33:17.698 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.698 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.698 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:17.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070253+0000", 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070307+0000", 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3909000000000002e-05 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070330+0000", 2024-04-24T14:33:17.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2599000000000002e-05 2024-04-24T14:33:17.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070337+0000", 2024-04-24T14:33:17.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3030000000000002e-06 2024-04-24T14:33:17.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070377+0000", 2024-04-24T14:33:17.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9916000000000002e-05 2024-04-24T14:33:17.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070398+0000", 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0579000000000001e-05 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073377+0000", 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002979013 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073427+0000", 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0078999999999999e-05 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.701 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.702 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:143 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'101, mlcod=13'98)", 2024-04-24T14:33:17.702 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:17.702 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.554814606999997, 2024-04-24T14:33:17.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030247590000000001, 2024-04-24T14:33:17.702 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.702 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.702 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:17.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076438+0000", 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076533+0000", 2024-04-24T14:33:17.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4443000000000004e-05 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076563+0000", 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0386999999999999e-05 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076604+0000", 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1012e-05 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076711+0000", 2024-04-24T14:33:17.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010637300000000001 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076737+0000", 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6190000000000002e-05 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079368+0000", 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026313529999999999 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079465+0000", 2024-04-24T14:33:17.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6426000000000001e-05 2024-04-24T14:33:17.706 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.706 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.706 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.706 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:159 1.0 e14/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'113, mlcod=14'109)", 2024-04-24T14:33:17.706 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:17.706 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.920994501000003, 2024-04-24T14:33:17.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052053589999999997, 2024-04-24T14:33:17.706 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.706 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.706 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710259+0000", 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710272+0000", 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.382e-05 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710303+0000", 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0629000000000001e-05 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710322+0000", 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8859999999999999e-05 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710479+0000", 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157076 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710535+0000", 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6289000000000003e-05 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715391+0000", 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048557139999999997 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715465+0000", 2024-04-24T14:33:17.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4301000000000006e-05 2024-04-24T14:33:17.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.710 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.710 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:161 1.0 e14/12 1:30306672:devicehealth::main.db.0000000000000000:head v 14'115, mlcod=14'112)", 2024-04-24T14:33:17.710 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:17.710 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.913883699000003, 2024-04-24T14:33:17.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039960450000000002, 2024-04-24T14:33:17.710 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:17.710 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717370+0000", 2024-04-24T14:33:17.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717499+0000", 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012877400000000001 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717513+0000", 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4229999999999999e-05 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717521+0000", 2024-04-24T14:33:17.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718040+0000", 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051920099999999995 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718094+0000", 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859000000000001e-05 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721278+0000", 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031838560000000001 2024-04-24T14:33:17.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:17.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:17.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:17.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721367+0000", 2024-04-24T14:33:17.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9241999999999997e-05 2024-04-24T14:33:17.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.714 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:17.714 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:17.714 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:17.715 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:17.630+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:17.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:17.728 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:17.730+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:17.819 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:17.820 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:17.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:17.830 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:17.831+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:17.929 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:17.930+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:17.930 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:17.930 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:17.930 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:17.930 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:17.930 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:17.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:18.030 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:18.035+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:18.040 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:18.040 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:18.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:18.040 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:18.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.040 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:1 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:18.040 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:18.040 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.840126197, 2024-04-24T14:33:18.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038534000000000002, 2024-04-24T14:33:18.041 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:18.041 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:18.041 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:18.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:18.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:18.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:18.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:18.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:18.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:18.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:18.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206414+0000", 2024-04-24T14:33:18.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2024-04-24T14:33:18.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:18.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206427+0000", 2024-04-24T14:33:18.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3457e-05 2024-04-24T14:33:18.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:18.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206450+0000", 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2966999999999999e-05 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206464+0000", 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3706999999999999e-05 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206581+0000", 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117161 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:18.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206802+0000", 2024-04-24T14:33:18.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221108 2024-04-24T14:33:18.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:18.044 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:18.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:18.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.044 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:18.044 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:18.044 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.8400499129999996, 2024-04-24T14:33:18.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056574800000000003, 2024-04-24T14:33:18.044 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:18.044 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:18.044 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:18.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:18.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:18.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:18.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:18.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:18.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:18.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:18.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206492+0000", 2024-04-24T14:33:18.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:18.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:18.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206497+0000", 2024-04-24T14:33:18.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6570000000000002e-06 2024-04-24T14:33:18.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:18.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206524+0000", 2024-04-24T14:33:18.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6503e-05 2024-04-24T14:33:18.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206535+0000", 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.092e-05 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206859+0000", 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032470600000000002 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207059+0000", 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000199132 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:18.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:3 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.8399476549999996, 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067768899999999998, 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:18.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:18.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:18.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:18.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206594+0000", 2024-04-24T14:33:18.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:18.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:18.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206601+0000", 2024-04-24T14:33:18.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7939999999999997e-06 2024-04-24T14:33:18.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:18.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206611+0000", 2024-04-24T14:33:18.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.008e-05 2024-04-24T14:33:18.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206617+0000", 2024-04-24T14:33:18.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4629999999999998e-06 2024-04-24T14:33:18.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:18.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207107+0000", 2024-04-24T14:33:18.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048966400000000001 2024-04-24T14:33:18.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:18.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:18.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:18.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207273+0000", 2024-04-24T14:33:18.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016570199999999999 2024-04-24T14:33:18.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:18.051 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:18.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:18.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:18.051 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:18.051 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:18.052 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:18.087 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5] 2024-04-24T14:33:18.087 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:33:18.087 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-04-24T14:33:18.087 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T14:33:18.087 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T14:33:18.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-04-24T14:33:18.131 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:18.134+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:18.160 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:18.160 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:18.170 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:18.199 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:18.199 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-24T14:33:18.199 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:18.232 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:18.235+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:18.272 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:18.272 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:18.281 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:18.332 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:18.334+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:18.385 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:18.385 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:18.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:18.386 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:18.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.386 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.386 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:18.386 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 56.06252499, 2024-04-24T14:33:18.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011830553000000001, 2024-04-24T14:33:18.386 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.386 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.386 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.386 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 36 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326000+0000", 2024-04-24T14:33:18.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:18.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326006+0000", 2024-04-24T14:33:18.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9340000000000003e-06 2024-04-24T14:33:18.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326044+0000", 2024-04-24T14:33:18.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8124999999999998e-05 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326054+0000", 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9650000000000003e-06 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332746+0000", 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006692539 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332833+0000", 2024-04-24T14:33:18.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6482000000000003e-05 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332947+0000", 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114472 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334320+0000", 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001372489 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337717+0000", 2024-04-24T14:33:18.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003396849 2024-04-24T14:33:18.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337789+0000", 2024-04-24T14:33:18.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2080000000000001e-05 2024-04-24T14:33:18.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337831+0000", 2024-04-24T14:33:18.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.244e-05 2024-04-24T14:33:18.391 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.391 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.391 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.392 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.392 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:18.392 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 56.062463127999997, 2024-04-24T14:33:18.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:33:18.392 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.392 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.392 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.392 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.392 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.392 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:33:18.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.392 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:33:18.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:33:18.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:33:18.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:33:18.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:33:18.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:33:18.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:33:18.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:33:18.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:33:18.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:33:18.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:33:18.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:33:18.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:33:18.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:33:18.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:33:18.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:33:18.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:33:18.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:33:18.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:33:18.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:33:18.397 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.397 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.397 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.398 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.398 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:18.398 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.968958655000002, 2024-04-24T14:33:18.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011218514000000001, 2024-04-24T14:33:18.398 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.398 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.398 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.398 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.398 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.398 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 88 2024-04-24T14:33:18.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.398 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:18.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:18.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419565+0000", 2024-04-24T14:33:18.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:18.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419575+0000", 2024-04-24T14:33:18.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5340000000000005e-06 2024-04-24T14:33:18.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419616+0000", 2024-04-24T14:33:18.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1301999999999999e-05 2024-04-24T14:33:18.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419638+0000", 2024-04-24T14:33:18.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2177000000000001e-05 2024-04-24T14:33:18.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428578+0000", 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089398280000000004 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428665+0000", 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6644000000000006e-05 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428797+0000", 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013231500000000001 2024-04-24T14:33:18.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.429866+0000", 2024-04-24T14:33:18.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001069221 2024-04-24T14:33:18.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430684+0000", 2024-04-24T14:33:18.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081788399999999995 2024-04-24T14:33:18.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430743+0000", 2024-04-24T14:33:18.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8875999999999997e-05 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430786+0000", 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2398999999999998e-05 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.762948186000003, 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:33:18.403 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.404 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.404 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.404 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.404 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.404 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:33:18.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.404 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:18.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:18.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:33:18.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:18.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:33:18.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:33:18.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:33:18.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:33:18.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:33:18.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:33:18.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:33:18.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:33:18.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:33:18.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:33:18.409 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.409 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.409 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.409 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.409 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:18.409 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.762722418000003, 2024-04-24T14:33:18.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:33:18.409 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.409 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.409 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.409 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:33:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:33:18.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:18.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:33:18.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:33:18.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:33:18.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:33:18.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:33:18.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:33:18.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:33:18.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:33:18.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:33:18.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:33:18.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:33:18.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:33:18.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:33:18.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:33:18.414 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.414 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.414 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.415 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.415 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:18.415 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.762545330999998, 2024-04-24T14:33:18.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:33:18.415 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.415 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.415 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.415 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.415 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.415 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:33:18.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.415 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:18.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:33:18.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:33:18.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:33:18.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:33:18.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:33:18.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:33:18.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:33:18.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:33:18.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:33:18.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:33:18.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.762441416999998, 2024-04-24T14:33:18.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:33:18.421 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.421 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.421 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.421 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.421 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.421 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:33:18.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.421 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:18.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:18.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:33:18.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:18.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:33:18.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:33:18.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.425 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.426 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.426 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.426 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:18.426 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.762145097000001, 2024-04-24T14:33:18.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:33:18.426 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.426 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.426 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:33:18.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.427 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:18.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:18.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:33:18.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:18.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:33:18.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:33:18.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:33:18.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:33:18.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:33:18.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:33:18.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:33:18.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:33:18.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:33:18.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.762052965999999, 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:18.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:18.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:33:18.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:18.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:33:18.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:33:18.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:33:18.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:33:18.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:33:18.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:33:18.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:33:18.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:33:18.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:33:18.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:33:18.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.437 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.438 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:18.438 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.761967753999997, 2024-04-24T14:33:18.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:33:18.438 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.438 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.438 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:33:18.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.438 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:18.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:18.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:33:18.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:18.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:33:18.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:33:18.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:33:18.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:33:18.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:33:18.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:33:18.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:33:18.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:33:18.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:33:18.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:33:18.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:33:18.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:33:18.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:33:18.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:33:18.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:33:18.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.444 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.444 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:18.444 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.761730872999998, 2024-04-24T14:33:18.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:33:18.444 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.444 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.444 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:33:18.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.444 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:33:18.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:33:18.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:33:18.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:33:18.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:33:18.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:33:18.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:33:18.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:33:18.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:33:18.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:33:18.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:33:18.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:33:18.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:33:18.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:33:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:33:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.761649351000003, 2024-04-24T14:33:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:33:18.449 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:18.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:33:18.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:18.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:33:18.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:33:18.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:33:18.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:33:18.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:33:18.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:33:18.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:33:18.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:33:18.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:33:18.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:33:18.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:33:18.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:33:18.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:33:18.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:33:18.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:33:18.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:33:18.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:33:18.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:33:18.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.329930275000002, 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01155111, 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.455 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.456 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135 2024-04-24T14:33:18.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.456 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:18.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:18.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:18.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:18.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058602+0000", 2024-04-24T14:33:18.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1239999999999999e-06 2024-04-24T14:33:18.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058609+0000", 2024-04-24T14:33:18.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6719999999999998e-06 2024-04-24T14:33:18.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058626+0000", 2024-04-24T14:33:18.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7569999999999999e-05 2024-04-24T14:33:18.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065271+0000", 2024-04-24T14:33:18.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00664469 2024-04-24T14:33:18.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065398+0000", 2024-04-24T14:33:18.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012694299999999999 2024-04-24T14:33:18.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065598+0000", 2024-04-24T14:33:18.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000200216 2024-04-24T14:33:18.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067596+0000", 2024-04-24T14:33:18.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019976450000000001 2024-04-24T14:33:18.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070055+0000", 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024593710000000001 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070109+0000", 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3690999999999998e-05 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070147+0000", 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7910000000000001e-05 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.460 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.329890939999999, 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:18.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:18.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:33:18.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:33:18.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:33:18.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:33:18.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:33:18.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:33:18.466 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.466 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.466 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.466 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.466 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:18.466 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.329849437, 2024-04-24T14:33:18.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:33:18.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:33:18.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:33:18.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:33:18.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:33:18.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:33:18.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:33:18.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:33:18.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:33:18.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:33:18.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:33:18.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:33:18.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:33:18.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:33:18.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:33:18.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:33:18.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.329737004000002, 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.472 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:33:18.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:18.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:33:18.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:33:18.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:33:18.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:33:18.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:33:18.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:33:18.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:33:18.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:33:18.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:33:18.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:33:18.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.478 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.478 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:18.478 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.329708703999998, 2024-04-24T14:33:18.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:33:18.478 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.478 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.478 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:33:18.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.478 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:33:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:33:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:33:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:33:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:33:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:33:18.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:33:18.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:33:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:33:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:33:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:33:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:33:18.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:33:18.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:33:18.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:33:18.483 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.483 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.483 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.483 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.483 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:18.483 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.329621000000003, 2024-04-24T14:33:18.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:33:18.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:33:18.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:33:18.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:33:18.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:33:18.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:33:18.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:33:18.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:33:18.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:33:18.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:33:18.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:33:18.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:33:18.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:33:18.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:33:18.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:33:18.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:33:18.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:33:18.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:33:18.489 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.489 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.490 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.490 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:18.490 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:18.490 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.329498199, 2024-04-24T14:33:18.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:33:18.491 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.491 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.491 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.491 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.491 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.491 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:33:18.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.491 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:18.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:18.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:33:18.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:18.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:33:18.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:33:18.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:33:18.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:33:18.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:33:18.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:33:18.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:33:18.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:33:18.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:33:18.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:33:18.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:33:18.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:33:18.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:33:18.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:33:18.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:33:18.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:33:18.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:33:18.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:33:18.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:33:18.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:33:18.499 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.500 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.500 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.500 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:33:18.500 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:18.500 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.686258266999999, 2024-04-24T14:33:18.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:33:18.501 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.501 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.501 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:18.501 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:18.501 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:18.501 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:33:18.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.502 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:18.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:18.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:33:18.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:18.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:33:18.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:33:18.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:33:18.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:33:18.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:33:18.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:33:18.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:33:18.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:33:18.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:33:18.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:33:18.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:18.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:33:18.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:33:18.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:18.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:33:18.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:33:18.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:18.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:33:18.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:33:18.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:33:18.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:33:18.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:33:18.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:33:18.511 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.511 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.511 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.511 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.511 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.511 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:18.513 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:18.514 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:18.514+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:18.614 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:18.615+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:18.661 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:18.661 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:18.662 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:18.662 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:18.671 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:18.715 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:18.719+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:18.771 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:18.772 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:18.772 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:18.772 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:18.772 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:18.781 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:18.816 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:18.818+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:18.883 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:0 3.1 e37/18 3:88c1567c:::rbd_trash_purge_schedule:head v 37'1, mlcod=0'0)", 2024-04-24T14:33:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.6819207650000001, 2024-04-24T14:33:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001532315, 2024-04-24T14:33:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:18.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208185+0000", 2024-04-24T14:33:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:18.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:18.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208203+0000", 2024-04-24T14:33:18.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.855e-05 2024-04-24T14:33:18.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:18.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208239+0000", 2024-04-24T14:33:18.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5299000000000002e-05 2024-04-24T14:33:18.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:18.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208253+0000", 2024-04-24T14:33:18.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3807e-05 2024-04-24T14:33:18.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:18.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208371+0000", 2024-04-24T14:33:18.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118921 2024-04-24T14:33:18.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:18.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208422+0000", 2024-04-24T14:33:18.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0917000000000001e-05 2024-04-24T14:33:18.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:18.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.209621+0000", 2024-04-24T14:33:18.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001198333 2024-04-24T14:33:18.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:18.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:18.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:18.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.209719+0000", 2024-04-24T14:33:18.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8717999999999999e-05 2024-04-24T14:33:18.888 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.888 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.888 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.888 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:18.888 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:18.888 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:18.893 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:18.917 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:18.919+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:19.018 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:19.023+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:19.037 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:19.037 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:19.037 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:19.037 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:19.046 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:19.119 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:19.122+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:19.147 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:19.147 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:19.147 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:19.147 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:19.147 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:19.156 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:19.220 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:19.223+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:19.302 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:19.302 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:19.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:19.302 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:19.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.302 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:1 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:19.302 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:19.302 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.1062338350000003, 2024-04-24T14:33:19.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002159392, 2024-04-24T14:33:19.302 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:19.303 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:19.303 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:19.303 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:19.303 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:19.303 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 1 2024-04-24T14:33:19.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.303 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:19.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:19.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:19.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:19.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:19.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:19.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:19.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:19.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201710+0000", 2024-04-24T14:33:19.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:33:19.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:19.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201731+0000", 2024-04-24T14:33:19.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0835999999999998e-05 2024-04-24T14:33:19.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201769+0000", 2024-04-24T14:33:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7472999999999998e-05 2024-04-24T14:33:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201796+0000", 2024-04-24T14:33:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6798999999999999e-05 2024-04-24T14:33:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201923+0000", 2024-04-24T14:33:19.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012770699999999999 2024-04-24T14:33:19.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:19.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203873+0000", 2024-04-24T14:33:19.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00194992 2024-04-24T14:33:19.306 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:19.306 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:19.306 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:19.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.306 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:19.306 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:19.307 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.1046680980000003, 2024-04-24T14:33:19.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026872300000000001, 2024-04-24T14:33:19.307 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:19.307 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:19.307 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:19.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:19.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:19.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:19.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:19.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:19.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:19.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:19.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203277+0000", 2024-04-24T14:33:19.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:19.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:19.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203292+0000", 2024-04-24T14:33:19.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4277000000000001e-05 2024-04-24T14:33:19.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203327+0000", 2024-04-24T14:33:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5219000000000003e-05 2024-04-24T14:33:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203350+0000", 2024-04-24T14:33:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2807e-05 2024-04-24T14:33:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203437+0000", 2024-04-24T14:33:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7070999999999994e-05 2024-04-24T14:33:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:19.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203548+0000", 2024-04-24T14:33:19.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000111576 2024-04-24T14:33:19.310 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:19.310 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:19.310 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:19.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.310 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:2 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:19.310 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:19.310 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.1045547930000001, 2024-04-24T14:33:19.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047788300000000002, 2024-04-24T14:33:19.310 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:19.311 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:19.311 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:19.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:19.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:19.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:19.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:19.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:19.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:19.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:19.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203391+0000", 2024-04-24T14:33:19.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:19.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:19.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203400+0000", 2024-04-24T14:33:19.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5860000000000005e-06 2024-04-24T14:33:19.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:19.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203421+0000", 2024-04-24T14:33:19.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1056e-05 2024-04-24T14:33:19.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:19.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203434+0000", 2024-04-24T14:33:19.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3093e-05 2024-04-24T14:33:19.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:19.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203623+0000", 2024-04-24T14:33:19.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000188646 2024-04-24T14:33:19.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:19.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203871+0000", 2024-04-24T14:33:19.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00024812199999999998 2024-04-24T14:33:19.314 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:19.314 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:19.314 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:19.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.314 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:19.314 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:19.314 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.1044973880000004, 2024-04-24T14:33:19.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059468100000000001, 2024-04-24T14:33:19.314 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:19.314 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203449+0000", 2024-04-24T14:33:19.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:19.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:19.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203469+0000", 2024-04-24T14:33:19.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0003000000000001e-05 2024-04-24T14:33:19.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:19.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203486+0000", 2024-04-24T14:33:19.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7309999999999999e-05 2024-04-24T14:33:19.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:19.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203501+0000", 2024-04-24T14:33:19.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.49e-05 2024-04-24T14:33:19.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:19.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203923+0000", 2024-04-24T14:33:19.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042195600000000002 2024-04-24T14:33:19.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:19.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204045+0000", 2024-04-24T14:33:19.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012202200000000001 2024-04-24T14:33:19.317 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:19.317 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:19.318 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:19.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.318 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:3 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:19.318 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:19.318 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.1031264600000004, 2024-04-24T14:33:19.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00093837399999999996, 2024-04-24T14:33:19.318 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:19.318 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:19.318 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:19.318 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:19.318 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:19.319 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 3 2024-04-24T14:33:19.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.319 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:19.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:19.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:19.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:19.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:19.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:19.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:19.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:19.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204819+0000", 2024-04-24T14:33:19.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:19.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:19.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204833+0000", 2024-04-24T14:33:19.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3346999999999999e-05 2024-04-24T14:33:19.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:19.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204866+0000", 2024-04-24T14:33:19.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3302000000000001e-05 2024-04-24T14:33:19.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:19.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204903+0000", 2024-04-24T14:33:19.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7576000000000001e-05 2024-04-24T14:33:19.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:19.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205040+0000", 2024-04-24T14:33:19.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013616 2024-04-24T14:33:19.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:19.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205231+0000", 2024-04-24T14:33:19.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019118999999999999 2024-04-24T14:33:19.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:19.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205610+0000", 2024-04-24T14:33:19.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037947000000000002 2024-04-24T14:33:19.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:19.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:19.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:19.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205760+0000", 2024-04-24T14:33:19.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000149259 2024-04-24T14:33:19.323 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:19.323 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:19.323 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:19.323 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:19.323 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:19.323 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:19.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:33:19.325 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:19.326+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:19.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:19.425 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:19.430+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:19.430 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:19.430 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:19.430 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:19.431 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:19.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:19.460 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:19.478 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:19.500 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:19.523 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:19.526 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:19.531+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:19.538 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:19.538 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:19.544 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:19.544 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:19.545 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:19.545 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:19.545 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:19.545 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:19.545 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:19.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:19.548 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:19.549 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:19.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T14:33:19.638 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:19.643+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:19.648 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:19.648 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:19.658 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:19.728 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:19.730+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:19.754 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:19.754 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:19.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:19.829 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:19.831+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:19.857 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:19.858 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:19.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:19.930 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:19.935+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:20.013 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:20.013 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:20.022 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:20.071 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:20.074+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:20.132 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:20.134+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:20.133 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:20.133 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:20.142 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:20.232 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:20.234+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:20.239 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:20.239 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:20.248 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:20.333 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:20.335+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:20.350 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:20.350 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:20.359 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:20.434 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:20.434+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:20.458 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:20.459 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:20.468 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:20.535 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:20.538+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:20.563 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:20.563 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:20.572 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:20.602 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-24T14:33:20.636 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:20.638+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:20.723 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:20.723 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:20.732 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:20.738 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:20.739+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:20.826 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:20.826 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:20.835 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:20.839 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:20.838+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:20.930 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:20.930 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:20.940 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:20.940 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:20.942+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:21.036 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:21.036 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:21.041 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:21.042+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:21.045 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:21.141 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:21.146+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:21.150 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:21.150 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:21.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:21.242 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:21.242+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:21.256 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:21.256 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:21.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:21.343 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:21.347+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:21.369 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:21.369 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:21.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:21.444 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:21.450+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:21.479 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:21.479 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:21.488 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:21.545 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:21.550+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:21.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:33:21.586 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:21.586 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:21.595 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:21.646 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:21.646+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:21.679 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:21.696 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:21.717 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:21.735 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:1 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.5354691700000007, 2024-04-24T14:33:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038534000000000002, 2024-04-24T14:33:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206414+0000", 2024-04-24T14:33:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2024-04-24T14:33:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206427+0000", 2024-04-24T14:33:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3457e-05 2024-04-24T14:33:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206450+0000", 2024-04-24T14:33:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2966999999999999e-05 2024-04-24T14:33:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206464+0000", 2024-04-24T14:33:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3706999999999999e-05 2024-04-24T14:33:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206581+0000", 2024-04-24T14:33:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117161 2024-04-24T14:33:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206802+0000", 2024-04-24T14:33:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221108 2024-04-24T14:33:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.5353928860000003, 2024-04-24T14:33:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056574800000000003, 2024-04-24T14:33:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206492+0000", 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206497+0000", 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6570000000000002e-06 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206524+0000", 2024-04-24T14:33:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6503e-05 2024-04-24T14:33:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206535+0000", 2024-04-24T14:33:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.092e-05 2024-04-24T14:33:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206859+0000", 2024-04-24T14:33:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032470600000000002 2024-04-24T14:33:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207059+0000", 2024-04-24T14:33:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000199132 2024-04-24T14:33:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:3 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.5352906280000003, 2024-04-24T14:33:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067768899999999998, 2024-04-24T14:33:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206594+0000", 2024-04-24T14:33:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206601+0000", 2024-04-24T14:33:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7939999999999997e-06 2024-04-24T14:33:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206611+0000", 2024-04-24T14:33:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.008e-05 2024-04-24T14:33:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206617+0000", 2024-04-24T14:33:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4629999999999998e-06 2024-04-24T14:33:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207107+0000", 2024-04-24T14:33:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048966400000000001 2024-04-24T14:33:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207273+0000", 2024-04-24T14:33:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016570199999999999 2024-04-24T14:33:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:21.747 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:21.748 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:21.748 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:21.749 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:21.750+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:21.765 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:21.766 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:21.766 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:21.766 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:21.766 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:21.766 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:21.766 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:21.769 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:21.769 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:21.849 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:21.851+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:21.859 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:21.859 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:21.859 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:21.859 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:21.870 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:21.949 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:21.950+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:21.971 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:21.971 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:21.971 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:21.971 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:21.971 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:21.980 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:22.051 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:22.054+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:22.081 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.758410324000003, 2024-04-24T14:33:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011830553000000001, 2024-04-24T14:33:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 36 2024-04-24T14:33:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326000+0000", 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326006+0000", 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9340000000000003e-06 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326044+0000", 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8124999999999998e-05 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326054+0000", 2024-04-24T14:33:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9650000000000003e-06 2024-04-24T14:33:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332746+0000", 2024-04-24T14:33:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006692539 2024-04-24T14:33:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332833+0000", 2024-04-24T14:33:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6482000000000003e-05 2024-04-24T14:33:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332947+0000", 2024-04-24T14:33:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114472 2024-04-24T14:33:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334320+0000", 2024-04-24T14:33:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001372489 2024-04-24T14:33:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337717+0000", 2024-04-24T14:33:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003396849 2024-04-24T14:33:22.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337789+0000", 2024-04-24T14:33:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2080000000000001e-05 2024-04-24T14:33:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337831+0000", 2024-04-24T14:33:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.244e-05 2024-04-24T14:33:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.088 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.088 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:22.088 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.758348462000001, 2024-04-24T14:33:22.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:33:22.088 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.088 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.088 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.088 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.088 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.088 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:33:22.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.088 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:22.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:22.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:33:22.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:22.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:33:22.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:33:22.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:33:22.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:33:22.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:33:22.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:33:22.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:33:22.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:33:22.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:33:22.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:33:22.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:33:22.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:33:22.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:33:22.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:33:22.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:33:22.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:33:22.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:33:22.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:33:22.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:33:22.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:33:22.093 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.093 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.093 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.093 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.094 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:22.094 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.664843988999998, 2024-04-24T14:33:22.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011218514000000001, 2024-04-24T14:33:22.094 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.094 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.094 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.094 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.094 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.094 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 88 2024-04-24T14:33:22.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.094 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:22.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:22.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419565+0000", 2024-04-24T14:33:22.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:22.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419575+0000", 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5340000000000005e-06 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419616+0000", 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1301999999999999e-05 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419638+0000", 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2177000000000001e-05 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428578+0000", 2024-04-24T14:33:22.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089398280000000004 2024-04-24T14:33:22.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428665+0000", 2024-04-24T14:33:22.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6644000000000006e-05 2024-04-24T14:33:22.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428797+0000", 2024-04-24T14:33:22.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013231500000000001 2024-04-24T14:33:22.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.429866+0000", 2024-04-24T14:33:22.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001069221 2024-04-24T14:33:22.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430684+0000", 2024-04-24T14:33:22.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081788399999999995 2024-04-24T14:33:22.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430743+0000", 2024-04-24T14:33:22.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8875999999999997e-05 2024-04-24T14:33:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430786+0000", 2024-04-24T14:33:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2398999999999998e-05 2024-04-24T14:33:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.458833519999999, 2024-04-24T14:33:22.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:33:22.100 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.100 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.100 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.100 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.100 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.100 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:33:22.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.100 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:22.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:22.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:33:22.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:22.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:33:22.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:33:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:33:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:33:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:33:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:33:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:33:22.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:33:22.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:33:22.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:33:22.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:33:22.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:33:22.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:33:22.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:33:22.105 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.105 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.105 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.105 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.105 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:22.105 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.458607751999999, 2024-04-24T14:33:22.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:33:22.105 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.105 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.105 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.105 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.106 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.106 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:33:22.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.106 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:22.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:22.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:33:22.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:22.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:33:22.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:33:22.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:33:22.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:33:22.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:33:22.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:33:22.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:33:22.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:33:22.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:33:22.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:33:22.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:33:22.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:33:22.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:33:22.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:33:22.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:33:22.113 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.113 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.113 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.113 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.113 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:22.113 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.458430665000002, 2024-04-24T14:33:22.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:33:22.113 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.113 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.113 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.113 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.113 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:33:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:33:22.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:22.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:33:22.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:33:22.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:33:22.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:33:22.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:33:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:33:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:33:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:33:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:33:22.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:33:22.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:33:22.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:33:22.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:33:22.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:33:22.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:33:22.118 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.118 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.118 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.119 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.119 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:22.119 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.458326751000001, 2024-04-24T14:33:22.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:33:22.119 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.119 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.119 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.119 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.119 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.119 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:33:22.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.119 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:22.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:33:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:33:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:33:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:33:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:33:22.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:33:22.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:33:22.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:33:22.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:33:22.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:33:22.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:33:22.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:33:22.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:33:22.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:33:22.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:33:22.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:33:22.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:33:22.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:33:22.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:33:22.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:33:22.124 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.124 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.124 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.125 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.125 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:22.125 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.458030430999997, 2024-04-24T14:33:22.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:33:22.125 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.125 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.125 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.125 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.125 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.125 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:33:22.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.125 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:22.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:22.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:33:22.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:33:22.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:33:22.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:33:22.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:33:22.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:33:22.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:33:22.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:33:22.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:33:22.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:33:22.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:33:22.130 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.130 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.130 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.130 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.130 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.457938300000002, 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:22.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:33:22.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:22.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:33:22.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:33:22.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:33:22.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:33:22.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:33:22.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:33:22.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:33:22.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:33:22.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:33:22.136 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.136 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.136 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.136 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.136 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:22.136 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.457853088, 2024-04-24T14:33:22.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:33:22.136 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.136 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.136 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.136 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.137 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.137 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:33:22.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.137 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:22.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:22.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:33:22.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:22.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:33:22.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:33:22.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:33:22.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:33:22.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:33:22.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:33:22.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:33:22.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:33:22.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:33:22.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:33:22.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:33:22.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:33:22.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:33:22.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:33:22.141 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.141 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.142 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.142 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.142 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:22.142 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.457616207000001, 2024-04-24T14:33:22.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:33:22.142 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.142 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.142 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.142 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.142 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.142 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:33:22.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:33:22.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:22.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:33:22.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:33:22.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:33:22.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:33:22.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:33:22.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:33:22.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:33:22.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:33:22.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:33:22.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:33:22.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:33:22.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:33:22.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:33:22.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:33:22.147 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.147 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.147 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.147 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.148 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:22.148 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.457534684999999, 2024-04-24T14:33:22.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:33:22.148 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.148 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.148 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.148 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.148 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.148 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:33:22.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.148 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:22.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:22.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:33:22.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:22.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:33:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:33:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:33:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:33:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:33:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:33:22.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:33:22.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:33:22.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:33:22.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:33:22.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:33:22.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:33:22.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:33:22.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:33:22.153 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.153 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.153 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.153 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.153 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:22.153 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.025815608999999, 2024-04-24T14:33:22.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01155111, 2024-04-24T14:33:22.154 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.154 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.154 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.154 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.154 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.154 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135 2024-04-24T14:33:22.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.154 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:22.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058602+0000", 2024-04-24T14:33:22.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1239999999999999e-06 2024-04-24T14:33:22.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058609+0000", 2024-04-24T14:33:22.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6719999999999998e-06 2024-04-24T14:33:22.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058626+0000", 2024-04-24T14:33:22.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7569999999999999e-05 2024-04-24T14:33:22.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065271+0000", 2024-04-24T14:33:22.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00664469 2024-04-24T14:33:22.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065398+0000", 2024-04-24T14:33:22.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012694299999999999 2024-04-24T14:33:22.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065598+0000", 2024-04-24T14:33:22.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000200216 2024-04-24T14:33:22.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067596+0000", 2024-04-24T14:33:22.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019976450000000001 2024-04-24T14:33:22.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070055+0000", 2024-04-24T14:33:22.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024593710000000001 2024-04-24T14:33:22.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070109+0000", 2024-04-24T14:33:22.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3690999999999998e-05 2024-04-24T14:33:22.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070147+0000", 2024-04-24T14:33:22.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7910000000000001e-05 2024-04-24T14:33:22.159 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.159 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.159 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.159 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.159 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:22.159 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.025776274000002, 2024-04-24T14:33:22.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:33:22.159 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.159 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:22.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:33:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:33:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:33:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:33:22.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:33:22.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:33:22.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:33:22.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:33:22.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:33:22.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:33:22.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:33:22.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:33:22.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:33:22.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:33:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:33:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.025734771000003, 2024-04-24T14:33:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:33:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.165 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.166 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.166 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.166 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:33:22.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.166 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:22.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:22.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:22.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:22.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:33:22.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:33:22.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:33:22.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:33:22.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:33:22.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:33:22.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:33:22.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:33:22.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:33:22.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:33:22.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:33:22.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:33:22.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:33:22.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:33:22.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:33:22.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:33:22.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:33:22.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:33:22.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:33:22.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:33:22.171 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.171 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.171 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.171 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.171 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:22.171 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.025622337999998, 2024-04-24T14:33:22.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:33:22.171 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.171 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.171 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.171 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.172 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.172 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:33:22.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.172 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:22.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:22.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:33:22.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:33:22.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:33:22.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:33:22.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:33:22.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:33:22.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:33:22.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:33:22.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:33:22.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:33:22.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:33:22.176 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.176 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.025594038000001, 2024-04-24T14:33:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:33:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:33:22.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.178 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:22.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:22.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:33:22.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:22.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:33:22.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:33:22.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:33:22.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:33:22.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:33:22.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:33:22.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:33:22.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:33:22.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:33:22.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:33:22.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:33:22.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:33:22.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:33:22.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:33:22.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:33:22.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:33:22.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:33:22.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:33:22.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:33:22.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:33:22.182 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.182 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.182 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.025506333999999, 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:22.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.184 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:22.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.184 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:33:22.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:22.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:33:22.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:33:22.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:33:22.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:33:22.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:33:22.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:33:22.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:33:22.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:33:22.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:33:22.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:33:22.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:33:22.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:33:22.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:33:22.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:33:22.188 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.188 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.188 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.188 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:22.188 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:22.188 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.025383533000003, 2024-04-24T14:33:22.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:33:22.189 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.189 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.189 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.189 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.189 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.189 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:33:22.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.189 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.189 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:22.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.190 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:22.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.190 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:33:22.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:22.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.190 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:33:22.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:33:22.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:33:22.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:33:22.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:33:22.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:33:22.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:33:22.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:33:22.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:33:22.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:33:22.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:33:22.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:33:22.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:33:22.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:33:22.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:33:22.194 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.194 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.194 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.194 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:33:22.194 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:22.194 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 58.382143601000003, 2024-04-24T14:33:22.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:33:22.194 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.194 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.194 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:33:22.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:33:22.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:22.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:33:22.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:33:22.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:22.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:33:22.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:33:22.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:22.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:33:22.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.199 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.200 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:22.201 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:22.201 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:22.206+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:22.302 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:22.303+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:22.350 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:22.350 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:22.350 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:22.359 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:22.402 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:22.407+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:22.455 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:22.456 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:22.456 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:22.456 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:22.456 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:22.464 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:22.503 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:22.507+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:22.562 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:0 3.1 e37/18 3:88c1567c:::rbd_trash_purge_schedule:head v 37'1, mlcod=0'0)", 2024-04-24T14:33:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.3611744370000007, 2024-04-24T14:33:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001532315, 2024-04-24T14:33:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208185+0000", 2024-04-24T14:33:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208203+0000", 2024-04-24T14:33:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.855e-05 2024-04-24T14:33:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208239+0000", 2024-04-24T14:33:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5299000000000002e-05 2024-04-24T14:33:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208253+0000", 2024-04-24T14:33:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3807e-05 2024-04-24T14:33:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208371+0000", 2024-04-24T14:33:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118921 2024-04-24T14:33:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208422+0000", 2024-04-24T14:33:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0917000000000001e-05 2024-04-24T14:33:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.209621+0000", 2024-04-24T14:33:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001198333 2024-04-24T14:33:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.209719+0000", 2024-04-24T14:33:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8717999999999999e-05 2024-04-24T14:33:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.567 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:22.571 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:22.604 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:22.606+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:22.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T14:33:22.673 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:22.674 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:22.674 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:22.674 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:22.683 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:22.727 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:22.730+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:22.782 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:22.782 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:22.782 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:22.782 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:22.782 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:22.791 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:22.806 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:22.811+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:22.890 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:22.890 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:22.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:22.890 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:22.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.890 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:1 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:22.890 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.6947673160000001, 2024-04-24T14:33:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002159392, 2024-04-24T14:33:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 1 2024-04-24T14:33:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:22.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201710+0000", 2024-04-24T14:33:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:33:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201731+0000", 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0835999999999998e-05 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201769+0000", 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7472999999999998e-05 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201796+0000", 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6798999999999999e-05 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201923+0000", 2024-04-24T14:33:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012770699999999999 2024-04-24T14:33:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203873+0000", 2024-04-24T14:33:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00194992 2024-04-24T14:33:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:22.894 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.6932015790000001, 2024-04-24T14:33:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026872300000000001, 2024-04-24T14:33:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203277+0000", 2024-04-24T14:33:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203292+0000", 2024-04-24T14:33:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4277000000000001e-05 2024-04-24T14:33:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203327+0000", 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5219000000000003e-05 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203350+0000", 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2807e-05 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203437+0000", 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7070999999999994e-05 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203548+0000", 2024-04-24T14:33:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000111576 2024-04-24T14:33:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:2 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.6930882740000008, 2024-04-24T14:33:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047788300000000002, 2024-04-24T14:33:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:22.898 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203391+0000", 2024-04-24T14:33:22.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203400+0000", 2024-04-24T14:33:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5860000000000005e-06 2024-04-24T14:33:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203421+0000", 2024-04-24T14:33:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1056e-05 2024-04-24T14:33:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203434+0000", 2024-04-24T14:33:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3093e-05 2024-04-24T14:33:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203623+0000", 2024-04-24T14:33:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000188646 2024-04-24T14:33:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203871+0000", 2024-04-24T14:33:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00024812199999999998 2024-04-24T14:33:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.901 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.6930308689999993, 2024-04-24T14:33:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059468100000000001, 2024-04-24T14:33:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:22.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203449+0000", 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203469+0000", 2024-04-24T14:33:22.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0003000000000001e-05 2024-04-24T14:33:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203486+0000", 2024-04-24T14:33:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7309999999999999e-05 2024-04-24T14:33:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203501+0000", 2024-04-24T14:33:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.49e-05 2024-04-24T14:33:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203923+0000", 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042195600000000002 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204045+0000", 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012202200000000001 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:3 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.6916599409999993, 2024-04-24T14:33:22.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00093837399999999996, 2024-04-24T14:33:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 3 2024-04-24T14:33:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204819+0000", 2024-04-24T14:33:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204833+0000", 2024-04-24T14:33:22.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3346999999999999e-05 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204866+0000", 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3302000000000001e-05 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204903+0000", 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7576000000000001e-05 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205040+0000", 2024-04-24T14:33:22.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013616 2024-04-24T14:33:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205231+0000", 2024-04-24T14:33:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019118999999999999 2024-04-24T14:33:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205610+0000", 2024-04-24T14:33:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037947000000000002 2024-04-24T14:33:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:22.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:22.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205760+0000", 2024-04-24T14:33:22.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000149259 2024-04-24T14:33:22.910 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.910 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.910 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.910 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:22.910 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:22.910 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:22.911 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:22.914+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:22.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:23.010 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:23.010 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:23.010 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:23.010 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:23.012 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:23.015+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:23.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:23.113 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:23.115+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:23.121 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:23.122 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:23.122 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:23.122 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:23.122 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:23.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:23.209 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5] 2024-04-24T14:33:23.209 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:33:23.209 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-24T14:33:23.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T14:33:23.213 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:23.215+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:23.282 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:23.283 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:23.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:23.283 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:23.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.283 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:36 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'22, mlcod=13'17)", 2024-04-24T14:33:23.283 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:23.283 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.954472977000002, 2024-04-24T14:33:23.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043791400000000001, 2024-04-24T14:33:23.283 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.283 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.283 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:23.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:23.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331851+0000", 2024-04-24T14:33:23.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331864+0000", 2024-04-24T14:33:23.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2767e-05 2024-04-24T14:33:23.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331881+0000", 2024-04-24T14:33:23.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6586e-05 2024-04-24T14:33:23.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331894+0000", 2024-04-24T14:33:23.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2976000000000001e-05 2024-04-24T14:33:23.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332161+0000", 2024-04-24T14:33:23.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026765399999999998 2024-04-24T14:33:23.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332197+0000", 2024-04-24T14:33:23.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5565999999999998e-05 2024-04-24T14:33:23.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336158+0000", 2024-04-24T14:33:23.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039616820000000002 2024-04-24T14:33:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336232+0000", 2024-04-24T14:33:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3151999999999994e-05 2024-04-24T14:33:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:37 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'23, mlcod=13'18)", 2024-04-24T14:33:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.953586498999996, 2024-04-24T14:33:23.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040417389999999999, 2024-04-24T14:33:23.288 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.288 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.288 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:23.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:23.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332738+0000", 2024-04-24T14:33:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332750+0000", 2024-04-24T14:33:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2432000000000001e-05 2024-04-24T14:33:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332768+0000", 2024-04-24T14:33:23.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7314000000000001e-05 2024-04-24T14:33:23.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332780+0000", 2024-04-24T14:33:23.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2716e-05 2024-04-24T14:33:23.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332925+0000", 2024-04-24T14:33:23.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144587 2024-04-24T14:33:23.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332955+0000", 2024-04-24T14:33:23.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0246000000000001e-05 2024-04-24T14:33:23.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336698+0000", 2024-04-24T14:33:23.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037430559999999998 2024-04-24T14:33:23.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336781+0000", 2024-04-24T14:33:23.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2447999999999994e-05 2024-04-24T14:33:23.291 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.291 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:39 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'25, mlcod=13'22)", 2024-04-24T14:33:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.947468176000001, 2024-04-24T14:33:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038736740000000001, 2024-04-24T14:33:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:23.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:23.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338856+0000", 2024-04-24T14:33:23.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338995+0000", 2024-04-24T14:33:23.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013868100000000001 2024-04-24T14:33:23.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339029+0000", 2024-04-24T14:33:23.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4372000000000002e-05 2024-04-24T14:33:23.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339050+0000", 2024-04-24T14:33:23.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0656999999999999e-05 2024-04-24T14:33:23.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339156+0000", 2024-04-24T14:33:23.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106177 2024-04-24T14:33:23.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339188+0000", 2024-04-24T14:33:23.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.243e-05 2024-04-24T14:33:23.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342648+0000", 2024-04-24T14:33:23.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034601689999999999 2024-04-24T14:33:23.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342731+0000", 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2534999999999995e-05 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:67 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'41, mlcod=13'39)", 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.896731891999998, 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031029009999999999, 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.296 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.297 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:23.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:23.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389592+0000", 2024-04-24T14:33:23.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389603+0000", 2024-04-24T14:33:23.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090000000000001e-05 2024-04-24T14:33:23.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389631+0000", 2024-04-24T14:33:23.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7772000000000001e-05 2024-04-24T14:33:23.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389646+0000", 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.432e-05 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389732+0000", 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6331999999999999e-05 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389771+0000", 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8751999999999998e-05 2024-04-24T14:33:23.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392585+0000", 2024-04-24T14:33:23.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028147469999999998 2024-04-24T14:33:23.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392696+0000", 2024-04-24T14:33:23.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111121 2024-04-24T14:33:23.300 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.300 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.300 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.895589280999999, 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030475519999999998, 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:23.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390735+0000", 2024-04-24T14:33:23.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390804+0000", 2024-04-24T14:33:23.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9488999999999998e-05 2024-04-24T14:33:23.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390832+0000", 2024-04-24T14:33:23.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7946e-05 2024-04-24T14:33:23.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390847+0000", 2024-04-24T14:33:23.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4693e-05 2024-04-24T14:33:23.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391508+0000", 2024-04-24T14:33:23.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066154799999999998 2024-04-24T14:33:23.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391538+0000", 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0029e-05 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393698+0000", 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002160052 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393784+0000", 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5414999999999995e-05 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.304 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.305 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.305 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:72 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'46, mlcod=13'43)", 2024-04-24T14:33:23.305 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:23.305 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.889415788000001, 2024-04-24T14:33:23.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036059460000000001, 2024-04-24T14:33:23.305 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.305 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.305 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:23.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:23.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396908+0000", 2024-04-24T14:33:23.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396968+0000", 2024-04-24T14:33:23.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9412e-05 2024-04-24T14:33:23.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397010+0000", 2024-04-24T14:33:23.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2169000000000002e-05 2024-04-24T14:33:23.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397022+0000", 2024-04-24T14:33:23.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1749999999999999e-05 2024-04-24T14:33:23.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397096+0000", 2024-04-24T14:33:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4701000000000002e-05 2024-04-24T14:33:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397126+0000", 2024-04-24T14:33:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2024-04-24T14:33:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400443+0000", 2024-04-24T14:33:23.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033172140000000002 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400516+0000", 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2181999999999994e-05 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:73 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'47, mlcod=13'43)", 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.888568384000003, 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055152170000000002, 2024-04-24T14:33:23.309 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.310 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.310 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:23.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:23.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397756+0000", 2024-04-24T14:33:23.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397884+0000", 2024-04-24T14:33:23.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012842100000000001 2024-04-24T14:33:23.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397923+0000", 2024-04-24T14:33:23.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8942000000000001e-05 2024-04-24T14:33:23.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397939+0000", 2024-04-24T14:33:23.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6073e-05 2024-04-24T14:33:23.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398106+0000", 2024-04-24T14:33:23.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016737100000000001 2024-04-24T14:33:23.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398137+0000", 2024-04-24T14:33:23.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0496000000000001e-05 2024-04-24T14:33:23.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403191+0000", 2024-04-24T14:33:23.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00505391 2024-04-24T14:33:23.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403272+0000", 2024-04-24T14:33:23.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1483999999999994e-05 2024-04-24T14:33:23.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.313 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.314 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:74 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'48, mlcod=13'43)", 2024-04-24T14:33:23.314 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:23.314 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.887769358, 2024-04-24T14:33:23.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048091760000000001, 2024-04-24T14:33:23.314 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.314 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.314 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:23.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398555+0000", 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398570+0000", 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.558e-05 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398610+0000", 2024-04-24T14:33:23.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0028999999999999e-05 2024-04-24T14:33:23.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398625+0000", 2024-04-24T14:33:23.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-04-24T14:33:23.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398701+0000", 2024-04-24T14:33:23.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6507000000000005e-05 2024-04-24T14:33:23.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398731+0000", 2024-04-24T14:33:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9356000000000002e-05 2024-04-24T14:33:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403281+0000", 2024-04-24T14:33:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004549796 2024-04-24T14:33:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403365+0000", 2024-04-24T14:33:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4685000000000007e-05 2024-04-24T14:33:23.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.318 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.318 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.318 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-04-24T14:33:23.318 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:23.318 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.866456433000003, 2024-04-24T14:33:23.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038140460000000002, 2024-04-24T14:33:23.318 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.318 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.318 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:23.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:23.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419868+0000", 2024-04-24T14:33:23.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419882+0000", 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.45e-05 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419919+0000", 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6939e-05 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419961+0000", 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2148999999999999e-05 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420084+0000", 2024-04-24T14:33:23.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122958 2024-04-24T14:33:23.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420122+0000", 2024-04-24T14:33:23.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8334999999999998e-05 2024-04-24T14:33:23.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423616+0000", 2024-04-24T14:33:23.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003493566 2024-04-24T14:33:23.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423683+0000", 2024-04-24T14:33:23.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7149000000000004e-05 2024-04-24T14:33:23.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.322 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.322 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T14:33:23.322 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:23.322 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.865044677999997, 2024-04-24T14:33:23.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030290730000000002, 2024-04-24T14:33:23.322 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.322 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.323 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:23.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:23.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421279+0000", 2024-04-24T14:33:23.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421352+0000", 2024-04-24T14:33:23.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2251999999999999e-05 2024-04-24T14:33:23.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421378+0000", 2024-04-24T14:33:23.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6006000000000001e-05 2024-04-24T14:33:23.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421388+0000", 2024-04-24T14:33:23.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0373e-05 2024-04-24T14:33:23.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421473+0000", 2024-04-24T14:33:23.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4665000000000003e-05 2024-04-24T14:33:23.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421512+0000", 2024-04-24T14:33:23.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9619000000000001e-05 2024-04-24T14:33:23.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424198+0000", 2024-04-24T14:33:23.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026854610000000001 2024-04-24T14:33:23.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424310+0000", 2024-04-24T14:33:23.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011208 2024-04-24T14:33:23.326 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.326 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.326 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.327 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:87 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'57, mlcod=13'51)", 2024-04-24T14:33:23.327 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:23.327 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.860781719000002, 2024-04-24T14:33:23.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002974057, 2024-04-24T14:33:23.327 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.327 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.327 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:23.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425542+0000", 2024-04-24T14:33:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425614+0000", 2024-04-24T14:33:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1328999999999994e-05 2024-04-24T14:33:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425664+0000", 2024-04-24T14:33:23.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0491999999999997e-05 2024-04-24T14:33:23.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425676+0000", 2024-04-24T14:33:23.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.134e-05 2024-04-24T14:33:23.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425798+0000", 2024-04-24T14:33:23.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012221700000000001 2024-04-24T14:33:23.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425860+0000", 2024-04-24T14:33:23.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1761999999999996e-05 2024-04-24T14:33:23.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428368+0000", 2024-04-24T14:33:23.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025084 2024-04-24T14:33:23.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428518+0000", 2024-04-24T14:33:23.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149884 2024-04-24T14:33:23.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.331 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.331 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-04-24T14:33:23.331 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:23.331 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.660672994000002, 2024-04-24T14:33:23.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002919694, 2024-04-24T14:33:23.331 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.331 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.331 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625651+0000", 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625667+0000", 2024-04-24T14:33:23.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5973000000000001e-05 2024-04-24T14:33:23.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625689+0000", 2024-04-24T14:33:23.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:33:23.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625705+0000", 2024-04-24T14:33:23.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5797000000000001e-05 2024-04-24T14:33:23.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625805+0000", 2024-04-24T14:33:23.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9238000000000001e-05 2024-04-24T14:33:23.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625837+0000", 2024-04-24T14:33:23.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2722000000000003e-05 2024-04-24T14:33:23.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628476+0000", 2024-04-24T14:33:23.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026391980000000002 2024-04-24T14:33:23.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628572+0000", 2024-04-24T14:33:23.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5754000000000005e-05 2024-04-24T14:33:23.335 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.335 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.335 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.335 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'88)", 2024-04-24T14:33:23.335 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:23.336 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.220735116, 2024-04-24T14:33:23.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031994179999999999, 2024-04-24T14:33:23.336 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.336 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.336 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:23.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065589+0000", 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065602+0000", 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2737e-05 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065620+0000", 2024-04-24T14:33:23.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8546000000000002e-05 2024-04-24T14:33:23.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065629+0000", 2024-04-24T14:33:23.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1600000000000004e-06 2024-04-24T14:33:23.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065729+0000", 2024-04-24T14:33:23.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9933999999999996e-05 2024-04-24T14:33:23.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065792+0000", 2024-04-24T14:33:23.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2712000000000005e-05 2024-04-24T14:33:23.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068667+0000", 2024-04-24T14:33:23.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028752920000000002 2024-04-24T14:33:23.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068790+0000", 2024-04-24T14:33:23.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012244400000000001 2024-04-24T14:33:23.339 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.339 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.340 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.340 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:137 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'91)", 2024-04-24T14:33:23.340 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:23.340 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.219379379999999, 2024-04-24T14:33:23.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052524399999999997, 2024-04-24T14:33:23.340 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.340 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.340 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:23.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:23.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066945+0000", 2024-04-24T14:33:23.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067031+0000", 2024-04-24T14:33:23.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6228000000000002e-05 2024-04-24T14:33:23.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067060+0000", 2024-04-24T14:33:23.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8445999999999998e-05 2024-04-24T14:33:23.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067069+0000", 2024-04-24T14:33:23.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999996e-06 2024-04-24T14:33:23.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067172+0000", 2024-04-24T14:33:23.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010285099999999999 2024-04-24T14:33:23.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067234+0000", 2024-04-24T14:33:23.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2068000000000003e-05 2024-04-24T14:33:23.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072090+0000", 2024-04-24T14:33:23.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048560859999999999 2024-04-24T14:33:23.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072199+0000", 2024-04-24T14:33:23.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108384 2024-04-24T14:33:23.344 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.344 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.344 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.344 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:138 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'96, mlcod=13'91)", 2024-04-24T14:33:23.344 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:23.344 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.218379616, 2024-04-24T14:33:23.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053159380000000001, 2024-04-24T14:33:23.345 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.345 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.345 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:23.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:23.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067945+0000", 2024-04-24T14:33:23.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067957+0000", 2024-04-24T14:33:23.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1963e-05 2024-04-24T14:33:23.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067982+0000", 2024-04-24T14:33:23.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5570000000000001e-05 2024-04-24T14:33:23.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067994+0000", 2024-04-24T14:33:23.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2306e-05 2024-04-24T14:33:23.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068282+0000", 2024-04-24T14:33:23.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028799000000000001 2024-04-24T14:33:23.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068307+0000", 2024-04-24T14:33:23.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4351000000000001e-05 2024-04-24T14:33:23.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073142+0000", 2024-04-24T14:33:23.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004835326 2024-04-24T14:33:23.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073262+0000", 2024-04-24T14:33:23.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119695 2024-04-24T14:33:23.348 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.348 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.349 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.349 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:139 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T14:33:23.349 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:23.349 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.216896789000003, 2024-04-24T14:33:23.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039447479999999997, 2024-04-24T14:33:23.349 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.349 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.349 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:23.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:23.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069428+0000", 2024-04-24T14:33:23.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:23.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069440+0000", 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2047e-05 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069469+0000", 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8852000000000001e-05 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069477+0000", 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069531+0000", 2024-04-24T14:33:23.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4126999999999999e-05 2024-04-24T14:33:23.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069559+0000", 2024-04-24T14:33:23.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7881e-05 2024-04-24T14:33:23.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073270+0000", 2024-04-24T14:33:23.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037104519999999999 2024-04-24T14:33:23.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073373+0000", 2024-04-24T14:33:23.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010380899999999999 2024-04-24T14:33:23.353 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.353 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.353 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.353 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:140 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'92)", 2024-04-24T14:33:23.353 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:23.353 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.216070913000003, 2024-04-24T14:33:23.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031722510000000001, 2024-04-24T14:33:23.353 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.354 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.354 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:23.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:23.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070253+0000", 2024-04-24T14:33:23.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:23.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070307+0000", 2024-04-24T14:33:23.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3909000000000002e-05 2024-04-24T14:33:23.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070330+0000", 2024-04-24T14:33:23.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2599000000000002e-05 2024-04-24T14:33:23.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070337+0000", 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3030000000000002e-06 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070377+0000", 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9916000000000002e-05 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070398+0000", 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0579000000000001e-05 2024-04-24T14:33:23.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073377+0000", 2024-04-24T14:33:23.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002979013 2024-04-24T14:33:23.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073427+0000", 2024-04-24T14:33:23.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0078999999999999e-05 2024-04-24T14:33:23.357 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.357 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.357 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.358 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:143 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'101, mlcod=13'98)", 2024-04-24T14:33:23.358 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:23.358 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.209885632999999, 2024-04-24T14:33:23.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030247590000000001, 2024-04-24T14:33:23.358 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.358 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.358 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:23.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:23.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076438+0000", 2024-04-24T14:33:23.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076533+0000", 2024-04-24T14:33:23.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4443000000000004e-05 2024-04-24T14:33:23.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076563+0000", 2024-04-24T14:33:23.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0386999999999999e-05 2024-04-24T14:33:23.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076604+0000", 2024-04-24T14:33:23.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1012e-05 2024-04-24T14:33:23.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076711+0000", 2024-04-24T14:33:23.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010637300000000001 2024-04-24T14:33:23.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076737+0000", 2024-04-24T14:33:23.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6190000000000002e-05 2024-04-24T14:33:23.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079368+0000", 2024-04-24T14:33:23.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026313529999999999 2024-04-24T14:33:23.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079465+0000", 2024-04-24T14:33:23.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6426000000000001e-05 2024-04-24T14:33:23.362 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.362 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.362 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:159 1.0 e14/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'113, mlcod=14'109)", 2024-04-24T14:33:23.362 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:23.362 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.576065526999997, 2024-04-24T14:33:23.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052053589999999997, 2024-04-24T14:33:23.362 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.362 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.363 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:23.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:23.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710259+0000", 2024-04-24T14:33:23.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:23.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710272+0000", 2024-04-24T14:33:23.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.382e-05 2024-04-24T14:33:23.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710303+0000", 2024-04-24T14:33:23.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0629000000000001e-05 2024-04-24T14:33:23.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710322+0000", 2024-04-24T14:33:23.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8859999999999999e-05 2024-04-24T14:33:23.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710479+0000", 2024-04-24T14:33:23.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157076 2024-04-24T14:33:23.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710535+0000", 2024-04-24T14:33:23.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6289000000000003e-05 2024-04-24T14:33:23.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715391+0000", 2024-04-24T14:33:23.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048557139999999997 2024-04-24T14:33:23.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715465+0000", 2024-04-24T14:33:23.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4301000000000006e-05 2024-04-24T14:33:23.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.366 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.367 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:161 1.0 e14/12 1:30306672:devicehealth::main.db.0000000000000000:head v 14'115, mlcod=14'112)", 2024-04-24T14:33:23.367 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:23.367 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.568954724999998, 2024-04-24T14:33:23.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039960450000000002, 2024-04-24T14:33:23.367 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.367 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:23.367 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:23.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:23.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717370+0000", 2024-04-24T14:33:23.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:23.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717499+0000", 2024-04-24T14:33:23.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012877400000000001 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717513+0000", 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4229999999999999e-05 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717521+0000", 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718040+0000", 2024-04-24T14:33:23.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051920099999999995 2024-04-24T14:33:23.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:23.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718094+0000", 2024-04-24T14:33:23.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859000000000001e-05 2024-04-24T14:33:23.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:23.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721278+0000", 2024-04-24T14:33:23.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031838560000000001 2024-04-24T14:33:23.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721367+0000", 2024-04-24T14:33:23.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9241999999999997e-05 2024-04-24T14:33:23.371 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.371 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.371 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.371 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.371 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.371 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:23.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:23.373 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:23.378+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:23.472 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:23.472 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:23.472 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:23.473 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:23.475+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:23.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:23.502 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:33:23.502 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 40, 2024-04-24T14:33:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "6706935e-d09a-472b-b839-321319fa9e54", 2024-04-24T14:33:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2024-04-24T14:32:09.059925+0000", 2024-04-24T14:33:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2024-04-24T14:33:20.584710+0000", 2024-04-24T14:33:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2024-04-24T14:33:00.483606+0000", 2024-04-24T14:33:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2024-04-24T14:32:11.347028+0000", 2024-04-24T14:33:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T14:33:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5801984, 2024-04-24T14:33:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2024-04-24T14:33:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "noscrub", 2024-04-24T14:33:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2024-04-24T14:33:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2024-04-24T14:33:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2024-04-24T14:33:23.503 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2024-04-24T14:33:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 4, 2024-04-24T14:33:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T14:33:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T14:33:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T14:33:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2024-04-24T14:33:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 3, 2024-04-24T14:33:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 6, 2024-04-24T14:33:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2024-04-24T14:33:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "jewel", 2024-04-24T14:33:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "quincy", 2024-04-24T14:33:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2024-04-24T14:33:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2024-04-24T14:33:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2024-04-24T14:33:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:21.233022+0000", 2024-04-24T14:33:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-24T14:33:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-24T14:33:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:33:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:33:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:33:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:33:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:33:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:33:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:33:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:33:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:33:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:33:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2024-04-24T14:33:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2024-04-24T14:33:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2024-04-24T14:33:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2024-04-24T14:33:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2024-04-24T14:33:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:33:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:33:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:33:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:33:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:33:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:33:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:33:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "14", 2024-04-24T14:33:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-24T14:33:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T14:33:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T14:33:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:33:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:33:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-24T14:33:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-24T14:33:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:33:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:33:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:33:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:33:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:33:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:33:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:33:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:33:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:33:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:33:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:33:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:33:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:33:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:33:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:33:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:33:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:33:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:33:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:33:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:33:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:33:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:33:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:33:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:33:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:33:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:33:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:33:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:33:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:33:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:33:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-24T14:33:23.509 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2024-04-24T14:33:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2024-04-24T14:33:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:33:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2024-04-24T14:33:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-24T14:33:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2024-04-24T14:33:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:26.609964+0000", 2024-04-24T14:33:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-24T14:33:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-24T14:33:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:33:23.510 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:33:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:33:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:33:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:33:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:33:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:33:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:33:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:33:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:33:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2024-04-24T14:33:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2024-04-24T14:33:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2024-04-24T14:33:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2024-04-24T14:33:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2024-04-24T14:33:23.511 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "21", 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "21", 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "21", 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "21", 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-24T14:33:23.512 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 19, 2024-04-24T14:33:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:33:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:33:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:33:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:33:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [ 2024-04-24T14:33:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: 3 2024-04-24T14:33:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:33:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": 3, 2024-04-24T14:33:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": 3, 2024-04-24T14:33:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:33:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:33:23.513 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:33:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:33:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:33:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:33:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:33:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:33:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:33:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:33:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:33:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:33:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:33:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:33:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:33:23.514 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:33:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:33:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:33:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:33:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:33:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:33:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:33:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:33:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:33:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:33:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-24T14:33:23.515 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "cache", 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:33.987564+0000", 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 9, 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,incomplete_clones", 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 4, 2024-04-24T14:33:23.516 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 4, 2024-04-24T14:33:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 4, 2024-04-24T14:33:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 4, 2024-04-24T14:33:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 4, 2024-04-24T14:33:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:33:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:33:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:33:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:33:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:33:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:33:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:33:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "25", 2024-04-24T14:33:23.517 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "21", 2024-04-24T14:33:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "21", 2024-04-24T14:33:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "21", 2024-04-24T14:33:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:33:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:33:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-24T14:33:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 19, 2024-04-24T14:33:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:33:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:33:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:33:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:33:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:33:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": 2, 2024-04-24T14:33:23.518 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:33:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:33:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "writeback", 2024-04-24T14:33:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:33:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 250, 2024-04-24T14:33:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:33:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:33:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:33:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:33:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:33:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:33:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:33:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bloom", 2024-04-24T14:33:23.519 INFO:teuthology.orchestra.run.smithi090.stdout: "false_positive_probability": 0.050000000000000003, 2024-04-24T14:33:23.520 INFO:teuthology.orchestra.run.smithi090.stdout: "target_size": 0, 2024-04-24T14:33:23.520 INFO:teuthology.orchestra.run.smithi090.stdout: "seed": 0 2024-04-24T14:33:23.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.520 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 60, 2024-04-24T14:33:23.520 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 8, 2024-04-24T14:33:23.520 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:33:23.520 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:33:23.520 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:33:23.520 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:33:23.520 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:33:23.520 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [ 2024-04-24T14:33:23.520 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:33:23.520 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:33:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:33:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:33:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:33:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:33:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:33:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000 2024-04-24T14:33:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:33:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:33:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:33:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:33:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:33:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:33:23.521 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-24T14:33:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-24T14:33:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "a2506b4f-3486-4f51-972f-1d9e4b015346", 2024-04-24T14:33:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:33:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:33:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:33:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:33:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:33:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:33:23.522 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:33:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 32, 2024-04-24T14:33:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:33:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:33:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:33:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2024-04-24T14:33:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.523 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2024-04-24T14:33:23.524 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:33:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2024-04-24T14:33:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.525 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2024-04-24T14:33:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:33:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2024-04-24T14:33:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.526 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2024-04-24T14:33:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6817/20309", 2024-04-24T14:33:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6819/20309", 2024-04-24T14:33:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/20309", 2024-04-24T14:33:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6821/20309", 2024-04-24T14:33:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:33:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:33:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:33:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-24T14:33:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "9e17324f-a615-494f-ba0d-7c8aa8dac95c", 2024-04-24T14:33:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 0, 2024-04-24T14:33:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:33:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:33:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:33:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:33:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:33:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:33:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 18, 2024-04-24T14:33:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 32, 2024-04-24T14:33:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:33:23.528 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:33:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2024-04-24T14:33:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2024-04-24T14:33:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.529 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:33:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2024-04-24T14:33:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2024-04-24T14:33:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:33:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2024-04-24T14:33:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2024-04-24T14:33:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.531 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:33:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2024-04-24T14:33:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2024-04-24T14:33:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:23.532 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/20308", 2024-04-24T14:33:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/20308", 2024-04-24T14:33:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/20308", 2024-04-24T14:33:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/20308", 2024-04-24T14:33:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:33:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "exists" 2024-04-24T14:33:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-24T14:33:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7", 2024-04-24T14:33:23.533 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:33:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:33:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:33:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:33:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:33:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:33:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:33:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 32, 2024-04-24T14:33:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:33:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:33:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:33:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2024-04-24T14:33:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2024-04-24T14:33:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:33:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2024-04-24T14:33:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2024-04-24T14:33:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:33:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2024-04-24T14:33:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2024-04-24T14:33:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:33:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2024-04-24T14:33:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2024-04-24T14:33:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6809/20310", 2024-04-24T14:33:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6811/20310", 2024-04-24T14:33:23.538 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6815/20310", 2024-04-24T14:33:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6813/20310", 2024-04-24T14:33:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:33:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:33:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:33:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-24T14:33:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "509c16da-22b2-48e8-aa51-c320c8c4fd8b", 2024-04-24T14:33:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:33:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:33:23.539 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:33:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:33:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:33:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:33:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:33:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 32, 2024-04-24T14:33:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:33:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:33:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:33:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6800", 2024-04-24T14:33:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:23.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6801", 2024-04-24T14:33:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:33:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6802", 2024-04-24T14:33:23.541 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6803", 2024-04-24T14:33:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:33:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6806", 2024-04-24T14:33:23.542 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6807", 2024-04-24T14:33:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:33:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6804", 2024-04-24T14:33:23.543 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6805", 2024-04-24T14:33:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6801/20316", 2024-04-24T14:33:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6803/20316", 2024-04-24T14:33:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6807/20316", 2024-04-24T14:33:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6805/20316", 2024-04-24T14:33:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:33:23.544 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1462066b-4e59-46d4-9b85-1f61ee1d56a6", 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 18, 2024-04-24T14:33:23.545 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:33:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:33:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:33:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6808", 2024-04-24T14:33:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6809", 2024-04-24T14:33:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:23.546 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:33:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6810", 2024-04-24T14:33:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6811", 2024-04-24T14:33:23.547 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:33:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6814", 2024-04-24T14:33:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6815", 2024-04-24T14:33:23.548 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:33:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6812", 2024-04-24T14:33:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6813", 2024-04-24T14:33:23.549 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6809/20318", 2024-04-24T14:33:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6811/20318", 2024-04-24T14:33:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6815/20318", 2024-04-24T14:33:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6813/20318", 2024-04-24T14:33:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:33:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:33:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:33:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-24T14:33:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "4dc954a8-5f3c-4826-b9bc-98cf30519382", 2024-04-24T14:33:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:33:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:33:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:33:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:33:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:33:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:33:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:33:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 32, 2024-04-24T14:33:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:33:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:33:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:33:23.551 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6816", 2024-04-24T14:33:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6817", 2024-04-24T14:33:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:33:23.552 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6818", 2024-04-24T14:33:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6819", 2024-04-24T14:33:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:33:23.553 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6822", 2024-04-24T14:33:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6823", 2024-04-24T14:33:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:33:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6820", 2024-04-24T14:33:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6821", 2024-04-24T14:33:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6817/20317", 2024-04-24T14:33:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6819/20317", 2024-04-24T14:33:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6823/20317", 2024-04-24T14:33:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6821/20317", 2024-04-24T14:33:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:33:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:33:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:33:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2024-04-24T14:33:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-24T14:33:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:33:23.556 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.792899+0000", 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.580414+0000", 2024-04-24T14:33:23.557 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 32 2024-04-24T14:33:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-24T14:33:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:33:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:33:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:33:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:33:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:33:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.713336+0000", 2024-04-24T14:33:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:33:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.558 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-24T14:33:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:33:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:33:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:33:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:33:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:33:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.579638+0000", 2024-04-24T14:33:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:33:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-24T14:33:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:33:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:33:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:33:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:33:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:33:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.570510+0000", 2024-04-24T14:33:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:33:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-24T14:33:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:33:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:33:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:33:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:33:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:33:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.736558+0000", 2024-04-24T14:33:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:33:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2024-04-24T14:33:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [], 2024-04-24T14:33:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2024-04-24T14:33:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2024-04-24T14:33:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2024-04-24T14:33:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2024-04-24T14:33:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2024-04-24T14:33:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2024-04-24T14:33:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-24T14:33:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-24T14:33:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-24T14:33:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-24T14:33:23.561 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2024-04-24T14:33:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [], 2024-04-24T14:33:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2024-04-24T14:33:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2024-04-24T14:33:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2024-04-24T14:33:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2024-04-24T14:33:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2024-04-24T14:33:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2024-04-24T14:33:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2024-04-24T14:33:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2024-04-24T14:33:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2024-04-24T14:33:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2024-04-24T14:33:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.563 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:23.564 INFO:tasks.thrashosds.thrasher:j is {'epoch': 40, 'fsid': '6706935e-d09a-472b-b839-321319fa9e54', 'created': '2024-04-24T14:32:09.059925+0000', 'modified': '2024-04-24T14:33:20.584710+0000', 'last_up_change': '2024-04-24T14:33:00.483606+0000', 'last_in_change': '2024-04-24T14:32:11.347028+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T14:32:21.233022+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T14:32:26.609964+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '21', 'last_force_op_resend_prenautilus': '21', 'last_force_op_resend_preluminous': '21', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-04-24T14:32:33.987564+0000', 'flags': 9, 'flags_names': 'hashpspool,incomplete_clones', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '21', 'last_force_op_resend_prenautilus': '21', 'last_force_op_resend_preluminous': '21', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 2, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'a2506b4f-3486-4f51-972f-1d9e4b015346', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 20309}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6819', 'nonce': 20309}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 20309}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6820', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 20309}]}, 'public_addr': '172.21.15.90:6817/20309', 'cluster_addr': '172.21.15.90:6819/20309', 'heartbeat_back_addr': '172.21.15.90:6823/20309', 'heartbeat_front_addr': '172.21.15.90:6821/20309', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9e17324f-a615-494f-ba0d-7c8aa8dac95c', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 32, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 20308}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 20308}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 20308}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 20308}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 20308}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 20308}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 20308}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 20308}]}, 'public_addr': '172.21.15.90:6801/20308', 'cluster_addr': '172.21.15.90:6803/20308', 'heartbeat_back_addr': '172.21.15.90:6807/20308', 'heartbeat_front_addr': '172.21.15.90:6805/20308', 'state': ['exists']}, {'osd': 2, 'uuid': 'f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 20310}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6810', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 20310}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6814', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 20310}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 20310}]}, 'public_addr': '172.21.15.90:6809/20310', 'cluster_addr': '172.21.15.90:6811/20310', 'heartbeat_back_addr': '172.21.15.90:6815/20310', 'heartbeat_front_addr': '172.21.15.90:6813/20310', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '509c16da-22b2-48e8-aa51-c320c8c4fd8b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6800', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6801', 'nonce': 20316}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6802', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6803', 'nonce': 20316}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6806', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6807', 'nonce': 20316}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6804', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6805', 'nonce': 20316}]}, 'public_addr': '172.21.15.103:6801/20316', 'cluster_addr': '172.21.15.103:6803/20316', 'heartbeat_back_addr': '172.21.15.103:6807/20316', 'heartbeat_front_addr': '172.21.15.103:6805/20316', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1462066b-4e59-46d4-9b85-1f61ee1d56a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6808', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6809', 'nonce': 20318}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6810', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6811', 'nonce': 20318}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6814', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6815', 'nonce': 20318}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6812', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6813', 'nonce': 20318}]}, 'public_addr': '172.21.15.103:6809/20318', 'cluster_addr': '172.21.15.103:6811/20318', 'heartbeat_back_addr': '172.21.15.103:6815/20318', 'heartbeat_front_addr': '172.21.15.103:6813/20318', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4dc954a8-5f3c-4826-b9bc-98cf30519382', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6816', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6817', 'nonce': 20317}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6818', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6819', 'nonce': 20317}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6822', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6823', 'nonce': 20317}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6820', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6821', 'nonce': 20317}]}, 'public_addr': '172.21.15.103:6817/20317', 'cluster_addr': '172.21.15.103:6819/20317', 'heartbeat_back_addr': '172.21.15.103:6823/20317', 'heartbeat_front_addr': '172.21.15.103:6821/20317', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.792899+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.580414+0000', 'dead_epoch': 32}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.713336+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.579638+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.570510+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.736558+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T14:33:23.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:33:23.574 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:23.575+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:23.580 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:23.580 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:23.580 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:23.580 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:23.580 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:23.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:23.608 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-24T14:33:23.675 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:23.678+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:23.705 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:23.705 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:23.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:23.705 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:23.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.705 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:1 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:23.705 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:23.705 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.505371154000001, 2024-04-24T14:33:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038534000000000002, 2024-04-24T14:33:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206414+0000", 2024-04-24T14:33:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2024-04-24T14:33:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206427+0000", 2024-04-24T14:33:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3457e-05 2024-04-24T14:33:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206450+0000", 2024-04-24T14:33:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2966999999999999e-05 2024-04-24T14:33:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206464+0000", 2024-04-24T14:33:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3706999999999999e-05 2024-04-24T14:33:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206581+0000", 2024-04-24T14:33:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117161 2024-04-24T14:33:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206802+0000", 2024-04-24T14:33:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221108 2024-04-24T14:33:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.50529487, 2024-04-24T14:33:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056574800000000003, 2024-04-24T14:33:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206492+0000", 2024-04-24T14:33:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206497+0000", 2024-04-24T14:33:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6570000000000002e-06 2024-04-24T14:33:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206524+0000", 2024-04-24T14:33:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6503e-05 2024-04-24T14:33:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206535+0000", 2024-04-24T14:33:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.092e-05 2024-04-24T14:33:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206859+0000", 2024-04-24T14:33:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032470600000000002 2024-04-24T14:33:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207059+0000", 2024-04-24T14:33:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000199132 2024-04-24T14:33:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:3 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.505192612, 2024-04-24T14:33:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067768899999999998, 2024-04-24T14:33:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206594+0000", 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206601+0000", 2024-04-24T14:33:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7939999999999997e-06 2024-04-24T14:33:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206611+0000", 2024-04-24T14:33:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.008e-05 2024-04-24T14:33:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206617+0000", 2024-04-24T14:33:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4629999999999998e-06 2024-04-24T14:33:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207107+0000", 2024-04-24T14:33:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048966400000000001 2024-04-24T14:33:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207273+0000", 2024-04-24T14:33:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016570199999999999 2024-04-24T14:33:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:23.716 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:23.717 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:23.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:33:23.775 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:23.779+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:23.828 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:23.828 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:23.828 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:23.828 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:23.837 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:23.859 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:33:23.862 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:33:23.872 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-04-24T14:33:23.609920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":19,"acting":19,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1795904,"kb_used_data":3572,"kb_used_omap":9,"kb_used_meta":1792246,"kb_avail":564435136,"statfs":{"total":579820584960,"available":577981579264,"internally_reserved":0,"allocated":3657728,"data_stored":2117474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1835259920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424607"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":31,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T14:33:18.408398+0000","last_change":"2024-04-24T14:32:28.321025+0000","last_active":"2024-04-24T14:33:18.408398+0000","last_peered":"2024-04-24T14:33:18.408398+0000","last_clean":"2024-04-24T14:33:18.408398+0000","last_became_active":"2024-04-24T14:32:28.320369+0000","last_became_peered":"2024-04-24T14:32:28.320369+0000","last_unstale":"2024-04-24T14:33:18.408398+0000","last_undegraded":"2024-04-24T14:33:18.408398+0000","last_fullsized":"2024-04-24T14:33:18.408398+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:49:07.246787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-04-24T14:33:18.408447+0000","last_change":"2024-04-24T14:33:01.511594+0000","last_active":"2024-04-24T14:33:18.408447+0000","last_peered":"2024-04-24T14:33:18.408447+0000","last_clean":"2024-04-24T14:32:27.283594+0000","last_became_active":"2024-04-24T14:33:01.511594+0000","last_became_peered":"2024-04-24T14:33:01.511594+0000","last_unstale":"2024-04-24T14:33:18.408447+0000","last_undegraded":"2024-04-24T14:33:18.408447+0000","last_fullsized":"2024-04-24T14:33:01.506313+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:14:40.168635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T14:33:18.408264+0000","last_change":"2024-04-24T14:32:28.321917+0000","last_active":"2024-04-24T14:33:18.408264+0000","last_peered":"2024-04-24T14:33:18.408264+0000","last_clean":"2024-04-24T14:33:18.408264+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:33:18.408264+0000","last_undegraded":"2024-04-24T14:33:18.408264+0000","last_fullsized":"2024-04-24T14:33:18.408264+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-04-24T14:33:17.583733+0000","last_change":"2024-04-24T14:33:01.502985+0000","last_active":"2024-04-24T14:33:17.583733+0000","last_peered":"2024-04-24T14:33:17.583733+0000","last_clean":"2024-04-24T14:32:27.283594+0000","last_became_active":"2024-04-24T14:33:01.502985+0000","last_became_peered":"2024-04-24T14:33:01.502985+0000","last_unstale":"2024-04-24T14:33:17.583733+0000","last_undegraded":"2024-04-24T14:33:17.583733+0000","last_fullsized":"2024-04-24T14:33:01.501533+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:22:56.636491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":28,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T14:33:17.760793+0000","last_change":"2024-04-24T14:32:35.360857+0000","last_active":"2024-04-24T14:33:17.760793+0000","last_peered":"2024-04-24T14:33:17.760793+0000","last_clean":"2024-04-24T14:33:17.760793+0000","last_became_active":"2024-04-24T14:32:35.360300+0000","last_became_peered":"2024-04-24T14:32:35.360300+0000","last_unstale":"2024-04-24T14:33:17.760793+0000","last_undegraded":"2024-04-24T14:33:17.760793+0000","last_fullsized":"2024-04-24T14:33:17.760793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:16:49.472230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":37,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-04-24T14:33:17.589730+0000","last_change":"2024-04-24T14:33:01.508881+0000","last_active":"2024-04-24T14:33:17.589730+0000","last_peered":"2024-04-24T14:33:17.589730+0000","last_clean":"2024-04-24T14:32:59.834073+0000","last_became_active":"2024-04-24T14:33:01.508881+0000","last_became_peered":"2024-04-24T14:33:01.508881+0000","last_unstale":"2024-04-24T14:33:17.589730+0000","last_undegraded":"2024-04-24T14:33:17.589730+0000","last_fullsized":"2024-04-24T14:33:01.507385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:47:05.753832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":16,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-04-24T14:33:17.584210+0000","last_change":"2024-04-24T14:33:01.501944+0000","last_active":"2024-04-24T14:33:17.584210+0000","last_peered":"2024-04-24T14:33:17.584210+0000","last_clean":"2024-04-24T14:32:34.325783+0000","last_became_active":"2024-04-24T14:33:01.501944+0000","last_became_peered":"2024-04-24T14:33:01.501944+0000","last_unstale":"2024-04-24T14:33:17.584210+0000","last_undegraded":"2024-04-24T14:33:17.584210+0000","last_fullsized":"2024-04-24T14:33:01.500778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:07:25.638522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-04-24T14:33:17.584360+0000","last_change":"2024-04-24T14:33:01.501910+0000","last_active":"2024-04-24T14:33:17.584360+0000","last_peered":"2024-04-24T14:33:17.584360+0000","last_clean":"2024-04-24T14:32:59.485776+0000","last_became_active":"2024-04-24T14:33:01.501910+0000","last_became_peered":"2024-04-24T14:33:01.501910+0000","last_unstale":"2024-04-24T14:33:17.584360+0000","last_undegraded":"2024-04-24T14:33:17.584360+0000","last_fullsized":"2024-04-24T14:33:01.500778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:29:31.384858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"37'1","reported_seq":30,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T14:33:17.583727+0000","last_change":"2024-04-24T14:32:35.360501+0000","last_active":"2024-04-24T14:33:17.583727+0000","last_peered":"2024-04-24T14:33:17.583727+0000","last_clean":"2024-04-24T14:33:17.583727+0000","last_became_active":"2024-04-24T14:32:35.359880+0000","last_became_peered":"2024-04-24T14:32:35.359880+0000","last_unstale":"2024-04-24T14:33:17.583727+0000","last_undegraded":"2024-04-24T14:33:17.583727+0000","last_fullsized":"2024-04-24T14:33:17.583727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:52:26.111730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":37,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-04-24T14:33:18.408377+0000","last_change":"2024-04-24T14:33:01.508797+0000","last_active":"2024-04-24T14:33:18.408377+0000","last_peered":"2024-04-24T14:33:18.408377+0000","last_clean":"2024-04-24T14:32:59.492028+0000","last_became_active":"2024-04-24T14:33:01.508797+0000","last_became_peered":"2024-04-24T14:33:01.508797+0000","last_unstale":"2024-04-24T14:33:18.408377+0000","last_undegraded":"2024-04-24T14:33:18.408377+0000","last_fullsized":"2024-04-24T14:33:01.506359+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:57:04.752690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"37'1","reported_seq":17,"reported_epoch":39,"state":"active+undersized+degraded","last_fresh":"2024-04-24T14:33:17.589667+0000","last_change":"2024-04-24T14:33:15.026785+0000","last_active":"2024-04-24T14:33:17.589667+0000","last_peered":"2024-04-24T14:33:17.589667+0000","last_clean":"2024-04-24T14:32:34.325783+0000","last_became_active":"2024-04-24T14:33:01.511527+0000","last_became_peered":"2024-04-24T14:33:01.511527+0000","last_unstale":"2024-04-24T14:33:17.589667+0000","last_undegraded":"2024-04-24T14:33:13.205459+0000","last_fullsized":"2024-04-24T14:33:01.507466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:45:22.561333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T14:33:17.589732+0000","last_change":"2024-04-24T14:32:28.322438+0000","last_active":"2024-04-24T14:33:17.589732+0000","last_peered":"2024-04-24T14:33:17.589732+0000","last_clean":"2024-04-24T14:33:17.589732+0000","last_became_active":"2024-04-24T14:32:28.321824+0000","last_became_peered":"2024-04-24T14:32:28.321824+0000","last_unstale":"2024-04-24T14:33:17.589732+0000","last_undegraded":"2024-04-24T14:33:17.589732+0000","last_fullsized":"2024-04-24T14:33:17.589732+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:26:37.494259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":184,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T14:33:18.408179+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:33:18.408179+0000","last_peered":"2024-04-24T14:33:18.408179+0000","last_clean":"2024-04-24T14:33:18.408179+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:33:18.408179+0000","last_undegraded":"2024-04-24T14:33:18.408179+0000","last_fullsized":"2024-04-24T14:33:18.408179+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":6,"acting":6,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640273,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297780,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074060,"statfs":{"total":96636764160,"available":96331837440,"internally_reserved":0,"allocated":430080,"data_stored":163547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.877},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.55600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":2,"last update":"Wed Apr 24 14:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Apr 24 14:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.67400000000000004}]},{"osd":4,"last update":"Wed Apr 24 14:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":1.7809999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":2.2240000000000002,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.82899999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640273,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297776,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074064,"statfs":{"total":96636764160,"available":96331841536,"internally_reserved":0,"allocated":430080,"data_stored":163547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.93500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":2,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":1.022}]},{"osd":3,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":0.77700000000000002}]},{"osd":5,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.81899999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640271,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302452,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94069388,"statfs":{"total":96636764160,"available":96327053312,"internally_reserved":0,"allocated":1024000,"data_stored":753915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]}]},{"osd":2,"up_from":11,"seq":47244640273,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297788,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074052,"statfs":{"total":96636764160,"available":96331829248,"internally_reserved":0,"allocated":442368,"data_stored":168470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.54900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":0.81599999999999995}]},{"osd":4,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.73599999999999999}]},{"osd":5,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.92200000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640268,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297656,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074184,"statfs":{"total":96636764160,"available":96331964416,"internally_reserved":0,"allocated":307200,"data_stored":114080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]}]},{"osd":0,"up_from":11,"seq":47244640272,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302452,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94069388,"statfs":{"total":96636764160,"available":96327053312,"internally_reserved":0,"allocated":1024000,"data_stored":753915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:33:23.873 INFO:tasks.thrashosds.thrasher:Setting 2.6 to [2, 1, 0, 5] 2024-04-24T14:33:23.873 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.6', '2', '1', '0', '5'] 2024-04-24T14:33:23.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.6 2 1 0 5 2024-04-24T14:33:23.876 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:23.878+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:23.907 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:23.926 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:23.940 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:23.941 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:23.941 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:23.941 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:23.941 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:23.949 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:23.950 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:23.969 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:23.991 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:23.991 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:23.992 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:23.992 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:23.992 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:23.992 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:23.992 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:23.995 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:23.995 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:23.995 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:23.998+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:24.047 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:24.047 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:24.056 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:24.078 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:24.083+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:24.166 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:24.179 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:24.182+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:24.260 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:24.260 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:24.269 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:24.280 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:24.283+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:24.381 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:24.382+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:24.413 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:24.413 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:24.423 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:24.482 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:24.483+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:24.517 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:24.517 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:24.526 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:24.583 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:24.587+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:24.619 INFO:teuthology.orchestra.run.smithi090.stderr:set 2.6 pg_upmap_items mapping to [2->1,0->5] 2024-04-24T14:33:24.628 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:24.628 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:24.637 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:24.683 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:24.686+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:24.735 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:24.735 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:24.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:24.784 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:24.786+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:24.861 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:24.861 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:24.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:24.885 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:24.890+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:24.976 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:24.976 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:24.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:24.986 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:24.986+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:25.082 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:25.082 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:25.086 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:25.086+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:25.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:25.187 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:25.191+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:25.188 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:25.189 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:25.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:25.288 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:25.291+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:25.340 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:25.340 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:25.349 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:25.389 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:25.390+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:25.445 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:25.445 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:25.454 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:25.490 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:25.495+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:25.564 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:25.564 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:25.574 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:25.592 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:25.594+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:25.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T14:33:25.675 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:25.675 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:25.684 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:25.693 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:25.695+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:25.781 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:25.781 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:25.791 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:25.793 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:25.795+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:25.892 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:25.892 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:25.894 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:25.898+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:25.901 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:25.995 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:25.995 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:25.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:33:25.996 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:25.999+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:26.004 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:26.096 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:26.098+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:26.099 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:26.099 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:26.108 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:26.137 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:26.157 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:26.179 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:26.197 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:26.199+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:26.198 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:26.209 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:26.209 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:26.218 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:26.226 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:26.226 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:26.226 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:26.227 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:26.227 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:26.227 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:26.227 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:26.231 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:26.298 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:26.303+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:26.314 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:26.314 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:26.314 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:26.314 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:26.314 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:26.323 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:26.400 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:26.402+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:26.423 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:26.423 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:1 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.228333397, 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002159392, 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 1 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.424 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201710+0000", 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:33:26.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:26.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201731+0000", 2024-04-24T14:33:26.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0835999999999998e-05 2024-04-24T14:33:26.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:26.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201769+0000", 2024-04-24T14:33:26.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7472999999999998e-05 2024-04-24T14:33:26.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201796+0000", 2024-04-24T14:33:26.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6798999999999999e-05 2024-04-24T14:33:26.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:26.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201923+0000", 2024-04-24T14:33:26.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012770699999999999 2024-04-24T14:33:26.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:26.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203873+0000", 2024-04-24T14:33:26.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00194992 2024-04-24T14:33:26.427 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:26.427 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:26.427 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:26.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.428 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:26.428 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:26.428 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.22676766, 2024-04-24T14:33:26.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026872300000000001, 2024-04-24T14:33:26.428 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:26.428 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:26.428 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:26.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:26.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:26.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:26.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203277+0000", 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203292+0000", 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4277000000000001e-05 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:26.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203327+0000", 2024-04-24T14:33:26.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5219000000000003e-05 2024-04-24T14:33:26.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203350+0000", 2024-04-24T14:33:26.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2807e-05 2024-04-24T14:33:26.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:26.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203437+0000", 2024-04-24T14:33:26.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7070999999999994e-05 2024-04-24T14:33:26.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:26.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203548+0000", 2024-04-24T14:33:26.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000111576 2024-04-24T14:33:26.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:26.431 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:26.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:26.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.431 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:2 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:26.431 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:26.431 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.226654355000001, 2024-04-24T14:33:26.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047788300000000002, 2024-04-24T14:33:26.431 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:26.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203391+0000", 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203400+0000", 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5860000000000005e-06 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203421+0000", 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1056e-05 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203434+0000", 2024-04-24T14:33:26.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3093e-05 2024-04-24T14:33:26.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:26.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203623+0000", 2024-04-24T14:33:26.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000188646 2024-04-24T14:33:26.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:26.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203871+0000", 2024-04-24T14:33:26.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00024812199999999998 2024-04-24T14:33:26.434 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.226596949999999, 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059468100000000001, 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:26.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:26.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:26.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:26.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:26.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:26.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203449+0000", 2024-04-24T14:33:26.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:26.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203469+0000", 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0003000000000001e-05 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203486+0000", 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7309999999999999e-05 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203501+0000", 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.49e-05 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:26.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203923+0000", 2024-04-24T14:33:26.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042195600000000002 2024-04-24T14:33:26.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:26.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204045+0000", 2024-04-24T14:33:26.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012202200000000001 2024-04-24T14:33:26.438 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:26.438 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:26.438 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:26.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.438 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:3 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:26.438 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:26.438 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.225226021999999, 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00093837399999999996, 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 3 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204819+0000", 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204833+0000", 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3346999999999999e-05 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204866+0000", 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3302000000000001e-05 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204903+0000", 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7576000000000001e-05 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205040+0000", 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013616 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:26.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205231+0000", 2024-04-24T14:33:26.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019118999999999999 2024-04-24T14:33:26.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:26.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205610+0000", 2024-04-24T14:33:26.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037947000000000002 2024-04-24T14:33:26.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:26.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:26.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:26.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205760+0000", 2024-04-24T14:33:26.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000149259 2024-04-24T14:33:26.442 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:26.442 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:26.443 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:26.443 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:26.443 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:26.443 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:26.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:26.501 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:26.506+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:26.551 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:26.551 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:26.551 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:26.552 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:26.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:26.654 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-24T14:33:26.655 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:26.658+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:26.756 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:26.762+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:26.760 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:26.760 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:26.760 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:26.770 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:26.856 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:26.859+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:26.871 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:26.871 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:26.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:26.872 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:26.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.872 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:36 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'22, mlcod=13'17)", 2024-04-24T14:33:26.872 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:26.872 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.544978037999996, 2024-04-24T14:33:26.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043791400000000001, 2024-04-24T14:33:26.872 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.872 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.872 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:26.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:26.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331851+0000", 2024-04-24T14:33:26.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331864+0000", 2024-04-24T14:33:26.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2767e-05 2024-04-24T14:33:26.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331881+0000", 2024-04-24T14:33:26.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6586e-05 2024-04-24T14:33:26.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331894+0000", 2024-04-24T14:33:26.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2976000000000001e-05 2024-04-24T14:33:26.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332161+0000", 2024-04-24T14:33:26.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026765399999999998 2024-04-24T14:33:26.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332197+0000", 2024-04-24T14:33:26.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5565999999999998e-05 2024-04-24T14:33:26.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336158+0000", 2024-04-24T14:33:26.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039616820000000002 2024-04-24T14:33:26.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336232+0000", 2024-04-24T14:33:26.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3151999999999994e-05 2024-04-24T14:33:26.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.876 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.876 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:37 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'23, mlcod=13'18)", 2024-04-24T14:33:26.876 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:26.876 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.544091559999998, 2024-04-24T14:33:26.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040417389999999999, 2024-04-24T14:33:26.877 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.877 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.877 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:26.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:26.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332738+0000", 2024-04-24T14:33:26.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:26.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332750+0000", 2024-04-24T14:33:26.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2432000000000001e-05 2024-04-24T14:33:26.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332768+0000", 2024-04-24T14:33:26.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7314000000000001e-05 2024-04-24T14:33:26.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332780+0000", 2024-04-24T14:33:26.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2716e-05 2024-04-24T14:33:26.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332925+0000", 2024-04-24T14:33:26.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144587 2024-04-24T14:33:26.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332955+0000", 2024-04-24T14:33:26.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0246000000000001e-05 2024-04-24T14:33:26.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336698+0000", 2024-04-24T14:33:26.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037430559999999998 2024-04-24T14:33:26.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336781+0000", 2024-04-24T14:33:26.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2447999999999994e-05 2024-04-24T14:33:26.881 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.881 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.881 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.881 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:39 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'25, mlcod=13'22)", 2024-04-24T14:33:26.881 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:26.881 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.537973237000003, 2024-04-24T14:33:26.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038736740000000001, 2024-04-24T14:33:26.881 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.882 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.882 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:26.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:26.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338856+0000", 2024-04-24T14:33:26.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338995+0000", 2024-04-24T14:33:26.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013868100000000001 2024-04-24T14:33:26.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339029+0000", 2024-04-24T14:33:26.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4372000000000002e-05 2024-04-24T14:33:26.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339050+0000", 2024-04-24T14:33:26.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0656999999999999e-05 2024-04-24T14:33:26.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339156+0000", 2024-04-24T14:33:26.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106177 2024-04-24T14:33:26.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339188+0000", 2024-04-24T14:33:26.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.243e-05 2024-04-24T14:33:26.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342648+0000", 2024-04-24T14:33:26.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034601689999999999 2024-04-24T14:33:26.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342731+0000", 2024-04-24T14:33:26.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2534999999999995e-05 2024-04-24T14:33:26.886 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.886 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.886 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.886 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:67 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'41, mlcod=13'39)", 2024-04-24T14:33:26.886 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:26.886 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.487236952999993, 2024-04-24T14:33:26.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031029009999999999, 2024-04-24T14:33:26.886 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.886 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.886 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:26.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:26.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389592+0000", 2024-04-24T14:33:26.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389603+0000", 2024-04-24T14:33:26.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090000000000001e-05 2024-04-24T14:33:26.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389631+0000", 2024-04-24T14:33:26.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7772000000000001e-05 2024-04-24T14:33:26.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389646+0000", 2024-04-24T14:33:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.432e-05 2024-04-24T14:33:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389732+0000", 2024-04-24T14:33:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6331999999999999e-05 2024-04-24T14:33:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389771+0000", 2024-04-24T14:33:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8751999999999998e-05 2024-04-24T14:33:26.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392585+0000", 2024-04-24T14:33:26.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028147469999999998 2024-04-24T14:33:26.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392696+0000", 2024-04-24T14:33:26.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111121 2024-04-24T14:33:26.890 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.890 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.890 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.486094342000001, 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030475519999999998, 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:26.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390735+0000", 2024-04-24T14:33:26.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390804+0000", 2024-04-24T14:33:26.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9488999999999998e-05 2024-04-24T14:33:26.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390832+0000", 2024-04-24T14:33:26.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7946e-05 2024-04-24T14:33:26.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390847+0000", 2024-04-24T14:33:26.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4693e-05 2024-04-24T14:33:26.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391508+0000", 2024-04-24T14:33:26.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066154799999999998 2024-04-24T14:33:26.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391538+0000", 2024-04-24T14:33:26.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0029e-05 2024-04-24T14:33:26.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393698+0000", 2024-04-24T14:33:26.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002160052 2024-04-24T14:33:26.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393784+0000", 2024-04-24T14:33:26.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5414999999999995e-05 2024-04-24T14:33:26.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.895 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.895 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:72 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'46, mlcod=13'43)", 2024-04-24T14:33:26.895 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:26.895 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.479920848999996, 2024-04-24T14:33:26.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036059460000000001, 2024-04-24T14:33:26.895 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.895 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.895 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:26.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:26.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396908+0000", 2024-04-24T14:33:26.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396968+0000", 2024-04-24T14:33:26.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9412e-05 2024-04-24T14:33:26.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397010+0000", 2024-04-24T14:33:26.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2169000000000002e-05 2024-04-24T14:33:26.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397022+0000", 2024-04-24T14:33:26.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1749999999999999e-05 2024-04-24T14:33:26.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397096+0000", 2024-04-24T14:33:26.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4701000000000002e-05 2024-04-24T14:33:26.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397126+0000", 2024-04-24T14:33:26.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2024-04-24T14:33:26.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400443+0000", 2024-04-24T14:33:26.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033172140000000002 2024-04-24T14:33:26.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400516+0000", 2024-04-24T14:33:26.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2181999999999994e-05 2024-04-24T14:33:26.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.899 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.900 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.900 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:73 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'47, mlcod=13'43)", 2024-04-24T14:33:26.900 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:26.900 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.479073444999997, 2024-04-24T14:33:26.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055152170000000002, 2024-04-24T14:33:26.900 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.900 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.900 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397756+0000", 2024-04-24T14:33:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397884+0000", 2024-04-24T14:33:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012842100000000001 2024-04-24T14:33:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397923+0000", 2024-04-24T14:33:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8942000000000001e-05 2024-04-24T14:33:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397939+0000", 2024-04-24T14:33:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6073e-05 2024-04-24T14:33:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398106+0000", 2024-04-24T14:33:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016737100000000001 2024-04-24T14:33:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398137+0000", 2024-04-24T14:33:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0496000000000001e-05 2024-04-24T14:33:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403191+0000", 2024-04-24T14:33:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00505391 2024-04-24T14:33:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403272+0000", 2024-04-24T14:33:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1483999999999994e-05 2024-04-24T14:33:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:74 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'48, mlcod=13'43)", 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.478274419000002, 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048091760000000001, 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398555+0000", 2024-04-24T14:33:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398570+0000", 2024-04-24T14:33:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.558e-05 2024-04-24T14:33:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398610+0000", 2024-04-24T14:33:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0028999999999999e-05 2024-04-24T14:33:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398625+0000", 2024-04-24T14:33:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-04-24T14:33:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398701+0000", 2024-04-24T14:33:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6507000000000005e-05 2024-04-24T14:33:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398731+0000", 2024-04-24T14:33:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9356000000000002e-05 2024-04-24T14:33:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403281+0000", 2024-04-24T14:33:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004549796 2024-04-24T14:33:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403365+0000", 2024-04-24T14:33:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4685000000000007e-05 2024-04-24T14:33:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-04-24T14:33:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.456961493999998, 2024-04-24T14:33:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038140460000000002, 2024-04-24T14:33:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419868+0000", 2024-04-24T14:33:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419882+0000", 2024-04-24T14:33:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.45e-05 2024-04-24T14:33:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419919+0000", 2024-04-24T14:33:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6939e-05 2024-04-24T14:33:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419961+0000", 2024-04-24T14:33:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2148999999999999e-05 2024-04-24T14:33:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420084+0000", 2024-04-24T14:33:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122958 2024-04-24T14:33:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420122+0000", 2024-04-24T14:33:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8334999999999998e-05 2024-04-24T14:33:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423616+0000", 2024-04-24T14:33:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003493566 2024-04-24T14:33:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423683+0000", 2024-04-24T14:33:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7149000000000004e-05 2024-04-24T14:33:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T14:33:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.455549739000006, 2024-04-24T14:33:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030290730000000002, 2024-04-24T14:33:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421279+0000", 2024-04-24T14:33:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421352+0000", 2024-04-24T14:33:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2251999999999999e-05 2024-04-24T14:33:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421378+0000", 2024-04-24T14:33:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6006000000000001e-05 2024-04-24T14:33:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421388+0000", 2024-04-24T14:33:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0373e-05 2024-04-24T14:33:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421473+0000", 2024-04-24T14:33:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4665000000000003e-05 2024-04-24T14:33:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421512+0000", 2024-04-24T14:33:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9619000000000001e-05 2024-04-24T14:33:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424198+0000", 2024-04-24T14:33:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026854610000000001 2024-04-24T14:33:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424310+0000", 2024-04-24T14:33:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011208 2024-04-24T14:33:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:87 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'57, mlcod=13'51)", 2024-04-24T14:33:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.451286780000004, 2024-04-24T14:33:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002974057, 2024-04-24T14:33:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425542+0000", 2024-04-24T14:33:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425614+0000", 2024-04-24T14:33:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1328999999999994e-05 2024-04-24T14:33:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425664+0000", 2024-04-24T14:33:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0491999999999997e-05 2024-04-24T14:33:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425676+0000", 2024-04-24T14:33:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.134e-05 2024-04-24T14:33:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425798+0000", 2024-04-24T14:33:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012221700000000001 2024-04-24T14:33:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425860+0000", 2024-04-24T14:33:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1761999999999996e-05 2024-04-24T14:33:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428368+0000", 2024-04-24T14:33:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025084 2024-04-24T14:33:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428518+0000", 2024-04-24T14:33:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149884 2024-04-24T14:33:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-04-24T14:33:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.251178054999997, 2024-04-24T14:33:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002919694, 2024-04-24T14:33:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625651+0000", 2024-04-24T14:33:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625667+0000", 2024-04-24T14:33:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5973000000000001e-05 2024-04-24T14:33:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625689+0000", 2024-04-24T14:33:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:33:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625705+0000", 2024-04-24T14:33:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5797000000000001e-05 2024-04-24T14:33:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625805+0000", 2024-04-24T14:33:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9238000000000001e-05 2024-04-24T14:33:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625837+0000", 2024-04-24T14:33:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2722000000000003e-05 2024-04-24T14:33:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628476+0000", 2024-04-24T14:33:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026391980000000002 2024-04-24T14:33:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628572+0000", 2024-04-24T14:33:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5754000000000005e-05 2024-04-24T14:33:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'88)", 2024-04-24T14:33:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.811240177000002, 2024-04-24T14:33:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031994179999999999, 2024-04-24T14:33:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065589+0000", 2024-04-24T14:33:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065602+0000", 2024-04-24T14:33:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2737e-05 2024-04-24T14:33:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065620+0000", 2024-04-24T14:33:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8546000000000002e-05 2024-04-24T14:33:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065629+0000", 2024-04-24T14:33:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1600000000000004e-06 2024-04-24T14:33:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065729+0000", 2024-04-24T14:33:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9933999999999996e-05 2024-04-24T14:33:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065792+0000", 2024-04-24T14:33:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2712000000000005e-05 2024-04-24T14:33:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068667+0000", 2024-04-24T14:33:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028752920000000002 2024-04-24T14:33:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068790+0000", 2024-04-24T14:33:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012244400000000001 2024-04-24T14:33:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:137 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'91)", 2024-04-24T14:33:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.809884441000001, 2024-04-24T14:33:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052524399999999997, 2024-04-24T14:33:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066945+0000", 2024-04-24T14:33:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067031+0000", 2024-04-24T14:33:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6228000000000002e-05 2024-04-24T14:33:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067060+0000", 2024-04-24T14:33:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8445999999999998e-05 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067069+0000", 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999996e-06 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067172+0000", 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010285099999999999 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067234+0000", 2024-04-24T14:33:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2068000000000003e-05 2024-04-24T14:33:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072090+0000", 2024-04-24T14:33:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048560859999999999 2024-04-24T14:33:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072199+0000", 2024-04-24T14:33:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108384 2024-04-24T14:33:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:138 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'96, mlcod=13'91)", 2024-04-24T14:33:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.808884677000002, 2024-04-24T14:33:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053159380000000001, 2024-04-24T14:33:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067945+0000", 2024-04-24T14:33:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067957+0000", 2024-04-24T14:33:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1963e-05 2024-04-24T14:33:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067982+0000", 2024-04-24T14:33:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5570000000000001e-05 2024-04-24T14:33:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067994+0000", 2024-04-24T14:33:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2306e-05 2024-04-24T14:33:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068282+0000", 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028799000000000001 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068307+0000", 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4351000000000001e-05 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073142+0000", 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004835326 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073262+0000", 2024-04-24T14:33:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119695 2024-04-24T14:33:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:139 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T14:33:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.807401849999998, 2024-04-24T14:33:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039447479999999997, 2024-04-24T14:33:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069428+0000", 2024-04-24T14:33:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069440+0000", 2024-04-24T14:33:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2047e-05 2024-04-24T14:33:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069469+0000", 2024-04-24T14:33:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8852000000000001e-05 2024-04-24T14:33:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069477+0000", 2024-04-24T14:33:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-04-24T14:33:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069531+0000", 2024-04-24T14:33:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4126999999999999e-05 2024-04-24T14:33:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069559+0000", 2024-04-24T14:33:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7881e-05 2024-04-24T14:33:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073270+0000", 2024-04-24T14:33:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037104519999999999 2024-04-24T14:33:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073373+0000", 2024-04-24T14:33:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010380899999999999 2024-04-24T14:33:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:140 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'92)", 2024-04-24T14:33:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.806575973999998, 2024-04-24T14:33:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031722510000000001, 2024-04-24T14:33:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070253+0000", 2024-04-24T14:33:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070307+0000", 2024-04-24T14:33:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3909000000000002e-05 2024-04-24T14:33:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070330+0000", 2024-04-24T14:33:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2599000000000002e-05 2024-04-24T14:33:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070337+0000", 2024-04-24T14:33:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3030000000000002e-06 2024-04-24T14:33:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070377+0000", 2024-04-24T14:33:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9916000000000002e-05 2024-04-24T14:33:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070398+0000", 2024-04-24T14:33:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0579000000000001e-05 2024-04-24T14:33:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073377+0000", 2024-04-24T14:33:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002979013 2024-04-24T14:33:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073427+0000", 2024-04-24T14:33:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0078999999999999e-05 2024-04-24T14:33:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:143 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'101, mlcod=13'98)", 2024-04-24T14:33:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.800390694000001, 2024-04-24T14:33:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030247590000000001, 2024-04-24T14:33:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076438+0000", 2024-04-24T14:33:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076533+0000", 2024-04-24T14:33:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4443000000000004e-05 2024-04-24T14:33:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076563+0000", 2024-04-24T14:33:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0386999999999999e-05 2024-04-24T14:33:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076604+0000", 2024-04-24T14:33:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1012e-05 2024-04-24T14:33:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076711+0000", 2024-04-24T14:33:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010637300000000001 2024-04-24T14:33:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076737+0000", 2024-04-24T14:33:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6190000000000002e-05 2024-04-24T14:33:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079368+0000", 2024-04-24T14:33:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026313529999999999 2024-04-24T14:33:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079465+0000", 2024-04-24T14:33:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6426000000000001e-05 2024-04-24T14:33:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:159 1.0 e14/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'113, mlcod=14'109)", 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.166570587999999, 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052053589999999997, 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710259+0000", 2024-04-24T14:33:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710272+0000", 2024-04-24T14:33:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.382e-05 2024-04-24T14:33:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710303+0000", 2024-04-24T14:33:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0629000000000001e-05 2024-04-24T14:33:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710322+0000", 2024-04-24T14:33:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8859999999999999e-05 2024-04-24T14:33:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710479+0000", 2024-04-24T14:33:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157076 2024-04-24T14:33:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710535+0000", 2024-04-24T14:33:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6289000000000003e-05 2024-04-24T14:33:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715391+0000", 2024-04-24T14:33:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048557139999999997 2024-04-24T14:33:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715465+0000", 2024-04-24T14:33:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4301000000000006e-05 2024-04-24T14:33:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:161 1.0 e14/12 1:30306672:devicehealth::main.db.0000000000000000:head v 14'115, mlcod=14'112)", 2024-04-24T14:33:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.159459785999999, 2024-04-24T14:33:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039960450000000002, 2024-04-24T14:33:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717370+0000", 2024-04-24T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717499+0000", 2024-04-24T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012877400000000001 2024-04-24T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717513+0000", 2024-04-24T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4229999999999999e-05 2024-04-24T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717521+0000", 2024-04-24T14:33:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2024-04-24T14:33:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718040+0000", 2024-04-24T14:33:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051920099999999995 2024-04-24T14:33:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718094+0000", 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859000000000001e-05 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721278+0000", 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031838560000000001 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721367+0000", 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9241999999999997e-05 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:26.964 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:26.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:26.965 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:26.967+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:27.066 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:27.071+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:27.111 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:27.111 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:27.112 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:27.112 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:27.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:27.167 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:27.170+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:27.221 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:27.222 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:27.222 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:27.222 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:27.222 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:27.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:27.268 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:27.270+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:27.333 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:27.333 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:27.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:1 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.133377477, 2024-04-24T14:33:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038534000000000002, 2024-04-24T14:33:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206414+0000", 2024-04-24T14:33:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2024-04-24T14:33:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206427+0000", 2024-04-24T14:33:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3457e-05 2024-04-24T14:33:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206450+0000", 2024-04-24T14:33:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2966999999999999e-05 2024-04-24T14:33:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206464+0000", 2024-04-24T14:33:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3706999999999999e-05 2024-04-24T14:33:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:27.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206581+0000", 2024-04-24T14:33:27.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117161 2024-04-24T14:33:27.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:27.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206802+0000", 2024-04-24T14:33:27.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221108 2024-04-24T14:33:27.337 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:27.337 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:27.337 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:27.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.337 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:27.338 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:27.338 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.133301192999999, 2024-04-24T14:33:27.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056574800000000003, 2024-04-24T14:33:27.338 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:27.338 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:27.338 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:27.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:27.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:27.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:27.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:27.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:27.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:27.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:27.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206492+0000", 2024-04-24T14:33:27.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:27.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:27.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206497+0000", 2024-04-24T14:33:27.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6570000000000002e-06 2024-04-24T14:33:27.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206524+0000", 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6503e-05 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206535+0000", 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.092e-05 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206859+0000", 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032470600000000002 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:27.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207059+0000", 2024-04-24T14:33:27.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000199132 2024-04-24T14:33:27.341 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:27.341 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:27.341 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:27.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.341 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:3 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:27.341 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:27.341 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.133198934999999, 2024-04-24T14:33:27.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067768899999999998, 2024-04-24T14:33:27.341 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:27.341 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:27.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206594+0000", 2024-04-24T14:33:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206601+0000", 2024-04-24T14:33:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7939999999999997e-06 2024-04-24T14:33:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206611+0000", 2024-04-24T14:33:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.008e-05 2024-04-24T14:33:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206617+0000", 2024-04-24T14:33:27.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4629999999999998e-06 2024-04-24T14:33:27.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:27.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207107+0000", 2024-04-24T14:33:27.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048966400000000001 2024-04-24T14:33:27.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:27.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:27.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:27.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207273+0000", 2024-04-24T14:33:27.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016570199999999999 2024-04-24T14:33:27.344 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:27.344 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:27.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:27.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:27.345 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:27.345 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:27.346 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:27.391 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:27.394+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:27.454 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:27.454 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:27.464 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:27.471 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:27.474+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:27.559 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:27.559 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:27.559 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:27.559 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:27.559 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:27.568 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:27.571 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:27.574+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:27.671 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:27.671 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:27.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:27.672 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:27.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.672 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.672 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:27.672 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.348399776999997, 2024-04-24T14:33:27.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011830553000000001, 2024-04-24T14:33:27.672 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.672 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.672 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.672 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.672 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.672 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 36 2024-04-24T14:33:27.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.673 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:27.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:27.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326000+0000", 2024-04-24T14:33:27.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:27.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326006+0000", 2024-04-24T14:33:27.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9340000000000003e-06 2024-04-24T14:33:27.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326044+0000", 2024-04-24T14:33:27.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8124999999999998e-05 2024-04-24T14:33:27.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326054+0000", 2024-04-24T14:33:27.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9650000000000003e-06 2024-04-24T14:33:27.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332746+0000", 2024-04-24T14:33:27.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006692539 2024-04-24T14:33:27.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332833+0000", 2024-04-24T14:33:27.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6482000000000003e-05 2024-04-24T14:33:27.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332947+0000", 2024-04-24T14:33:27.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114472 2024-04-24T14:33:27.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334320+0000", 2024-04-24T14:33:27.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001372489 2024-04-24T14:33:27.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337717+0000", 2024-04-24T14:33:27.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003396849 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337789+0000", 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2080000000000001e-05 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337831+0000", 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.244e-05 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.677 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.678 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:27.678 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.348337915000002, 2024-04-24T14:33:27.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:33:27.678 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.678 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.678 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.678 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.678 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.678 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:33:27.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.678 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:27.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:27.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:33:27.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:27.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:33:27.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:33:27.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:33:27.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:33:27.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:33:27.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:33:27.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:33:27.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:33:27.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:33:27.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:33:27.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:33:27.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.254833442000006, 2024-04-24T14:33:27.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011218514000000001, 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 88 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:27.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419565+0000", 2024-04-24T14:33:27.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:27.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419575+0000", 2024-04-24T14:33:27.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5340000000000005e-06 2024-04-24T14:33:27.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419616+0000", 2024-04-24T14:33:27.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1301999999999999e-05 2024-04-24T14:33:27.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419638+0000", 2024-04-24T14:33:27.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2177000000000001e-05 2024-04-24T14:33:27.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428578+0000", 2024-04-24T14:33:27.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089398280000000004 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428665+0000", 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6644000000000006e-05 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428797+0000", 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013231500000000001 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.429866+0000", 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001069221 2024-04-24T14:33:27.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430684+0000", 2024-04-24T14:33:27.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081788399999999995 2024-04-24T14:33:27.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430743+0000", 2024-04-24T14:33:27.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8875999999999997e-05 2024-04-24T14:33:27.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430786+0000", 2024-04-24T14:33:27.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2398999999999998e-05 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.048822973, 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:33:27.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.690 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:27.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:27.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:33:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:33:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:33:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:33:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:33:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:33:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:33:27.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:33:27.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:33:27.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:33:27.694 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.694 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.694 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.694 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.048597204999993, 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:27.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:33:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:33:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:33:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:33:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:33:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:33:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:33:27.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:33:27.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:33:27.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:33:27.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:33:27.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.048420117999996, 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.700 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:33:27.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:27.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.702 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.702 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:33:27.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:33:27.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.702 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.702 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:33:27.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:33:27.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.702 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:33:27.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:33:27.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.704 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:33:27.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:33:27.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.704 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:33:27.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:33:27.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.048316204000002, 2024-04-24T14:33:27.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:33:27.706 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.706 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.706 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.706 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.706 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.706 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:33:27.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.706 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:27.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:33:27.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:33:27.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:33:27.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:33:27.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:33:27.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:33:27.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:33:27.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:33:27.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:33:27.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:33:27.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:33:27.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:33:27.711 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.711 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.711 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.711 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.711 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:27.711 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.048019883999999, 2024-04-24T14:33:27.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:33:27.711 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.711 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.711 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.711 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.711 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.713 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.714 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:33:27.714 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:33:27.714 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.714 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.714 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.714 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:33:27.714 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:33:27.714 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.714 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.714 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.714 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:33:27.714 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:33:27.714 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.716 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.047927752999996, 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:27.718 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.718 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:33:27.718 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:27.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.718 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:33:27.718 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:33:27.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.719 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.719 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.719 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:33:27.719 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:33:27.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.719 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.719 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.719 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:33:27.719 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:33:27.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.719 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.719 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.719 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:33:27.719 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:33:27.720 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:33:27.721 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.047842540999994, 2024-04-24T14:33:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:33:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.722 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.723 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:33:27.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.723 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:27.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:27.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.724 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.724 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:33:27.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:27.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.724 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.724 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.724 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:33:27.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:33:27.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.724 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.724 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.724 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:33:27.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:33:27.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.725 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.725 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.725 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:33:27.725 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:33:27.725 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.725 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.725 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.725 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:33:27.725 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:33:27.725 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.725 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.726 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:33:27.726 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:33:27.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.726 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:33:27.726 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:33:27.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.726 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:33:27.726 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:33:27.727 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.047605660000002, 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:33:27.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:33:27.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:33:27.730 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:33:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:33:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.047524138, 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.733 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.734 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.734 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.734 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:33:27.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.734 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.734 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:27.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.734 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:27.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:33:27.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.736 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.737 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.737 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:33:27.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:33:27.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.737 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.737 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:33:27.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:33:27.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.737 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.737 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:33:27.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:33:27.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:33:27.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:33:27.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:33:27.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:33:27.738 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.738 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.738 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.739 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.739 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:27.739 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.615805062000007, 2024-04-24T14:33:27.739 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01155111, 2024-04-24T14:33:27.739 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.739 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.739 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.739 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.739 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.739 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135 2024-04-24T14:33:27.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.739 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.740 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058602+0000", 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1239999999999999e-06 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058609+0000", 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6719999999999998e-06 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058626+0000", 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7569999999999999e-05 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.742 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065271+0000", 2024-04-24T14:33:27.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00664469 2024-04-24T14:33:27.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.742 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065398+0000", 2024-04-24T14:33:27.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012694299999999999 2024-04-24T14:33:27.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.742 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065598+0000", 2024-04-24T14:33:27.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000200216 2024-04-24T14:33:27.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067596+0000", 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019976450000000001 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070055+0000", 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024593710000000001 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070109+0000", 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3690999999999998e-05 2024-04-24T14:33:27.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070147+0000", 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7910000000000001e-05 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.615765726999996, 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.744 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:27.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:33:27.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:27.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:33:27.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:33:27.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:33:27.749 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:33:27.749 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.749 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.749 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:33:27.749 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:33:27.749 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.749 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.749 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:33:27.749 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:33:27.749 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.749 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.750 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.750 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.750 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.750 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:27.750 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.615724224000004, 2024-04-24T14:33:27.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:33:27.750 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.750 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.750 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.750 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.750 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.750 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:33:27.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:27.751 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:27.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.752 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.752 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.752 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:33:27.752 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:33:27.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.752 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.752 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.752 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:33:27.752 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:33:27.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.752 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.752 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.752 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.753 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:33:27.754 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:33:27.754 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.754 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.754 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.754 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:33:27.754 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:33:27.754 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.754 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.754 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.754 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:33:27.754 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:33:27.754 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.754 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.755 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.615611791000006, 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.756 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.757 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:27.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.757 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:33:27.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:27.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.757 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:33:27.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:33:27.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:33:27.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:33:27.759 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.760 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.760 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.760 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:33:27.760 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:33:27.760 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.760 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.760 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.760 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:33:27.760 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:33:27.760 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.760 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.760 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.760 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:33:27.760 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:33:27.761 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.761 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.761 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.761 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.761 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.761 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:27.761 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.615583490999995, 2024-04-24T14:33:27.761 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:33:27.761 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.761 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.761 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.761 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.761 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:33:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.762 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.763 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:33:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:33:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:33:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:33:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:33:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:33:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.766 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.766 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:33:27.766 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:33:27.766 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.766 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.766 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.766 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:33:27.766 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:33:27.766 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.766 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.766 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.766 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.766 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.766 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.615495787, 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:27.767 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:33:27.768 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.769 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.770 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.770 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:33:27.770 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:33:27.770 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.770 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.770 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.770 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:33:27.770 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:33:27.770 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.770 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.770 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.770 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:33:27.770 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:33:27.771 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.771 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.771 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.771 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:33:27.771 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:33:27.771 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.771 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.771 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.771 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:33:27.771 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:33:27.771 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.771 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.771 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.771 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.615372985999997, 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.772 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.773 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:33:27.773 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.773 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.773 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.773 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.773 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:27.773 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.773 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.773 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.773 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.773 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:27.773 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.773 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.773 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.774 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.774 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:33:27.774 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:27.774 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.774 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.774 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.774 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:33:27.774 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:33:27.774 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.774 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.774 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.774 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:33:27.774 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:33:27.775 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.776 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.776 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.776 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:33:27.776 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:33:27.776 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.776 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.776 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.776 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:33:27.776 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:33:27.776 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.776 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.776 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.776 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:33:27.776 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.777 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.778 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:33:27.778 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:27.778 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 63.972133053999997, 2024-04-24T14:33:27.778 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:33:27.778 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:27.778 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:27.778 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:27.778 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:27.778 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:27.778 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:33:27.778 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.778 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:27.778 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.778 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.779 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:27.780 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:33:27.780 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:33:27.780 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.780 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.780 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:27.780 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:33:27.780 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:33:27.780 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.780 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.780 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:27.780 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:33:27.780 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:33:27.781 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.782 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.782 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:27.782 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:33:27.782 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:33:27.782 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.782 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.782 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:27.782 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:33:27.782 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:33:27.782 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.782 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.782 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:27.782 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:33:27.782 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:33:27.783 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:27.783 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:27.783 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:27.783 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:33:27.783 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:33:27.783 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.783 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.783 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.783 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:27.783 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:27.783 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:27.785 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:27.785 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:27.791+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:27.885 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:27.886+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:27.933 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:27.933 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:27.933 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:27.933 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:27.942 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:27.987 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:27.990+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:28.040 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:28.041 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:28.041 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:28.041 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:28.041 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:28.050 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:28.087 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:28.090+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:28.147 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:28.147 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:28.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:28.147 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:28.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:28.147 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:0 3.1 e37/18 3:88c1567c:::rbd_trash_purge_schedule:head v 37'1, mlcod=0'0)", 2024-04-24T14:33:28.147 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:28.147 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 14.945456317, 2024-04-24T14:33:28.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001532315, 2024-04-24T14:33:28.147 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:28.147 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:28.147 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:28.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208185+0000", 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:28.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208203+0000", 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.855e-05 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208239+0000", 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5299000000000002e-05 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208253+0000", 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3807e-05 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:28.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:28.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208371+0000", 2024-04-24T14:33:28.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118921 2024-04-24T14:33:28.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:28.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:28.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:28.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208422+0000", 2024-04-24T14:33:28.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0917000000000001e-05 2024-04-24T14:33:28.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:28.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:28.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:28.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.209621+0000", 2024-04-24T14:33:28.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001198333 2024-04-24T14:33:28.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:28.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:28.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:28.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.209719+0000", 2024-04-24T14:33:28.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8717999999999999e-05 2024-04-24T14:33:28.151 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:28.151 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:28.151 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:28.151 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:28.151 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:28.151 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:28.156 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:28.188 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:28.190+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:28.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:33:28.252 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:28.252 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:28.252 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:28.252 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:28.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:28.289 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:28.291+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:28.374 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:28.390 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:28.391+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:28.394 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:28.405 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:28.405 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:28.414 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:28.417 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:28.441 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:28.464 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:28.464 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:28.464 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:28.465 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:28.465 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:28.465 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:28.465 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:28.467 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:28.490 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:28.494+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:28.513 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:28.514 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:28.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:28.592 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:28.594+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:28.629 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:28.629 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:28.639 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:28.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T14:33:28.693 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:28.695+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:28.751 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:28.752 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:28.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:28.806 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:28.811+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:28.853 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:28.853 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:28.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:28.895 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:28.899+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:28.959 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:28.959 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:28.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:28.995 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:28.999+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:29.064 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:29.064 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:29.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:29.097 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:29.098+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:29.197 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:29.199+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:29.219 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:29.219 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:29.228 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:29.297 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:29.299+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:29.337 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:29.337 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:29.347 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:29.399 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:29.403+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:29.442 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:29.442 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:29.452 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:29.500 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:29.503+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:29.551 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:29.551 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:29.560 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:29.600 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:29.602+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:29.631 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5] 2024-04-24T14:33:29.632 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:33:29.632 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5] 2024-04-24T14:33:29.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-04-24T14:33:29.661 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-24T14:33:29.662 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:29.662 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:29.671 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:29.718 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:29.723+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:29.767 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:29.767 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:29.777 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:29.802 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:29.802+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:29.875 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:29.875 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:29.883 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:29.903 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:29.906+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:29.934 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-04-24T14:33:29.935+0000 7ff9c25de700 -1 mon.a@0(leader).osd e44 definitely_dead 0 2024-04-24T14:33:29.979 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:29.979 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:29.988 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:30.004 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:30.007+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:30.085 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:30.085 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:30.094 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:30.143 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:30.146+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:30.192 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:30.192 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:30.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:30.205 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:30.206+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:30.305 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:30.307+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:30.308 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:30.308 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:30.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:30.406 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:30.407+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:30.418 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:30.418 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:30.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:30.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:33:30.506 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:30.511+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:30.572 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:30.572 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:30.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:30.607 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:30.611+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:30.619 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:30.656 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:30.665 INFO:teuthology.orchestra.run.smithi090.stderr:marked out osd.5. 2024-04-24T14:33:30.682 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:30.690 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:30.690 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:30.690 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:30.690 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:30.699 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:30.702 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:30.708 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:30.710+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:30.732 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:30.733 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:30.733 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:30.733 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:30.733 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:30.733 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:30.733 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:30.737 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:30.737 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:30.795 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:30.795 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:30.796 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:30.796 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:30.796 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:30.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:30.808 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:30.810+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:30.843 INFO:tasks.workunit.client.0.smithi173.stderr:Note: switching to '330067dbce47655d1f762ef14d546d2fde16a6a9'. 2024-04-24T14:33:30.843 INFO:tasks.workunit.client.0.smithi173.stderr: 2024-04-24T14:33:30.843 INFO:tasks.workunit.client.0.smithi173.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-24T14:33:30.843 INFO:tasks.workunit.client.0.smithi173.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-24T14:33:30.843 INFO:tasks.workunit.client.0.smithi173.stderr:state without impacting any branches by switching back to a branch. 2024-04-24T14:33:30.843 INFO:tasks.workunit.client.0.smithi173.stderr: 2024-04-24T14:33:30.843 INFO:tasks.workunit.client.0.smithi173.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-24T14:33:30.843 INFO:tasks.workunit.client.0.smithi173.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-24T14:33:30.843 INFO:tasks.workunit.client.0.smithi173.stderr: 2024-04-24T14:33:30.843 INFO:tasks.workunit.client.0.smithi173.stderr: git switch -c 2024-04-24T14:33:30.843 INFO:tasks.workunit.client.0.smithi173.stderr: 2024-04-24T14:33:30.843 INFO:tasks.workunit.client.0.smithi173.stderr:Or undo this operation with: 2024-04-24T14:33:30.843 INFO:tasks.workunit.client.0.smithi173.stderr: 2024-04-24T14:33:30.844 INFO:tasks.workunit.client.0.smithi173.stderr: git switch - 2024-04-24T14:33:30.844 INFO:tasks.workunit.client.0.smithi173.stderr: 2024-04-24T14:33:30.844 INFO:tasks.workunit.client.0.smithi173.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-24T14:33:30.844 INFO:tasks.workunit.client.0.smithi173.stderr: 2024-04-24T14:33:30.844 INFO:tasks.workunit.client.0.smithi173.stderr:HEAD is now at 330067dbce4 Merge PR #54172 into wip-yuriw-testing-20240423.151325-quincy 2024-04-24T14:33:30.848 DEBUG:teuthology.orchestra.run.smithi173:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-24T14:33:30.878 INFO:tasks.workunit.client.0.smithi173.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-24T14:33:30.881 INFO:tasks.workunit.client.0.smithi173.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-24T14:33:30.881 INFO:tasks.workunit.client.0.smithi173.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-24T14:33:30.904 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:30.904 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:30.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:30.904 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:1 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.704792935, 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038534000000000002, 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:30.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:30.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:30.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:30.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206414+0000", 2024-04-24T14:33:30.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2024-04-24T14:33:30.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:30.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206427+0000", 2024-04-24T14:33:30.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3457e-05 2024-04-24T14:33:30.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206450+0000", 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2966999999999999e-05 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206464+0000", 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3706999999999999e-05 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206581+0000", 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117161 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206802+0000", 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221108 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.704716650999998, 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056574800000000003, 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:30.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206492+0000", 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:30.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206497+0000", 2024-04-24T14:33:30.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6570000000000002e-06 2024-04-24T14:33:30.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:30.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206524+0000", 2024-04-24T14:33:30.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6503e-05 2024-04-24T14:33:30.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:30.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206535+0000", 2024-04-24T14:33:30.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.092e-05 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206859+0000", 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032470600000000002 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207059+0000", 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000199132 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:3 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.704614393, 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067768899999999998, 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:30.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206594+0000", 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206601+0000", 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7939999999999997e-06 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:30.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206611+0000", 2024-04-24T14:33:30.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.008e-05 2024-04-24T14:33:30.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:30.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206617+0000", 2024-04-24T14:33:30.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4629999999999998e-06 2024-04-24T14:33:30.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:30.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207107+0000", 2024-04-24T14:33:30.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048966400000000001 2024-04-24T14:33:30.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:30.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:30.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:30.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207273+0000", 2024-04-24T14:33:30.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016570199999999999 2024-04-24T14:33:30.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:30.915 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:30.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:30.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:30.915 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:30.915 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:30.916 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:30.917 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:30.919+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:31.017 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:31.018+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:31.025 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:31.025 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:31.026 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:31.026 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:31.035 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:31.118 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:31.118+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:31.136 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:31.136 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:31.136 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:31.136 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:31.136 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:31.145 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:31.219 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:31.223+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:31.245 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:31.245 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:31.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:31.246 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:31.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.246 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.246 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:31.246 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.921878883999995, 2024-04-24T14:33:31.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011830553000000001, 2024-04-24T14:33:31.246 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 36 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326000+0000", 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326006+0000", 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9340000000000003e-06 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326044+0000", 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8124999999999998e-05 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326054+0000", 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9650000000000003e-06 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332746+0000", 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006692539 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332833+0000", 2024-04-24T14:33:31.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6482000000000003e-05 2024-04-24T14:33:31.250 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.250 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.250 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332947+0000", 2024-04-24T14:33:31.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114472 2024-04-24T14:33:31.250 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.250 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.250 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334320+0000", 2024-04-24T14:33:31.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001372489 2024-04-24T14:33:31.250 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.250 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337717+0000", 2024-04-24T14:33:31.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003396849 2024-04-24T14:33:31.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.251 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337789+0000", 2024-04-24T14:33:31.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2080000000000001e-05 2024-04-24T14:33:31.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.251 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337831+0000", 2024-04-24T14:33:31.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.244e-05 2024-04-24T14:33:31.251 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.921817021999999, 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:33:31.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:31.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.254 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:33:31.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:33:31.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.254 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:33:31.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:33:31.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.254 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.255 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:33:31.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.828312549000003, 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011218514000000001, 2024-04-24T14:33:31.257 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.258 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.258 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.258 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.258 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.258 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 88 2024-04-24T14:33:31.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.258 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.258 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.258 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:31.258 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.258 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419567+0000", 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419565+0000", 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419575+0000", 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.5340000000000005e-06 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.260 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419616+0000", 2024-04-24T14:33:31.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1301999999999999e-05 2024-04-24T14:33:31.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.260 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.419638+0000", 2024-04-24T14:33:31.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2177000000000001e-05 2024-04-24T14:33:31.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.260 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428578+0000", 2024-04-24T14:33:31.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089398280000000004 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428665+0000", 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6644000000000006e-05 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.428797+0000", 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013231500000000001 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.429866+0000", 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001069221 2024-04-24T14:33:31.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.262 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430684+0000", 2024-04-24T14:33:31.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00081788399999999995 2024-04-24T14:33:31.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.262 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430743+0000", 2024-04-24T14:33:31.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8875999999999997e-05 2024-04-24T14:33:31.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.262 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.430786+0000", 2024-04-24T14:33:31.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2398999999999998e-05 2024-04-24T14:33:31.263 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.263 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.263 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.263 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.263 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:31.263 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.622302079999997, 2024-04-24T14:33:31.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:33:31.263 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.263 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.263 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.263 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:33:31.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:33:31.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.266 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.266 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:33:31.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:33:31.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.266 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.266 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:33:31.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:33:31.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.267 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:31.268 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.622076312000004, 2024-04-24T14:33:31.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:33:31.269 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.269 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.269 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.269 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.269 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.269 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:33:31.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.269 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:31.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:31.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:33:31.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:31.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:33:31.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:33:31.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:33:31.274 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.274 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.274 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.274 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.274 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:31.274 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.621899225000007, 2024-04-24T14:33:31.274 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:33:31.274 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.274 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.275 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.275 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.275 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.275 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:33:31.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.275 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:31.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:31.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.276 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.276 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:33:31.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:31.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:33:31.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:33:31.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.278 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:33:31.278 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:33:31.278 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.278 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.278 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.278 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:33:31.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.621795311, 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:33:31.281 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:31.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:33:31.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.284 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.284 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:33:31.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:33:31.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.284 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.284 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:33:31.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:33:31.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.286 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.287 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.287 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.287 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:31.287 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.621498990999996, 2024-04-24T14:33:31.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:33:31.287 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.287 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.287 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.287 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.287 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.287 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:33:31.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.288 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:31.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:31.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.289 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.289 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:33:31.289 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:31.289 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.289 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.289 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:33:31.290 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:33:31.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.290 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.290 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.290 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:33:31.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:33:31.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.291 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.291 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:33:31.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:33:31.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:33:31.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:33:31.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:33:31.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:33:31.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:33:31.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:33:31.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:33:31.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:33:31.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.294 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.294 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:33:31.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:33:31.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:33:31.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:33:31.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.296 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:33:31.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:33:31.296 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.296 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.296 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.296 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.296 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:31.296 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.621406859999993, 2024-04-24T14:33:31.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:33:31.296 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.297 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.297 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.297 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.297 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.297 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:33:31.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.297 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:31.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:31.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.300 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:33:31.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:31.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.300 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.300 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:33:31.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:33:31.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.300 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.300 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:33:31.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:33:31.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.301 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.301 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:33:31.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:33:31.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:33:31.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:33:31.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:33:31.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:33:31.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:33:31.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:33:31.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:33:31.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:33:31.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:33:31.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:33:31.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:33:31.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:33:31.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:33:31.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:33:31.306 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.306 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.306 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.307 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.307 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:31.307 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.621321648000006, 2024-04-24T14:33:31.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:33:31.307 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.308 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.308 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.308 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.308 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.308 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:33:31.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.308 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:31.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:31.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:33:31.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:31.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:33:31.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:33:31.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:33:31.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:33:31.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:33:31.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:33:31.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:33:31.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:33:31.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:33:31.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:33:31.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:33:31.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:33:31.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:33:31.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:33:31.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.315 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.316 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.316 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.316 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:31.316 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.621084766999999, 2024-04-24T14:33:31.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:33:31.316 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.316 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.316 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.316 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.316 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.316 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:33:31.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.317 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:31.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:31.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:33:31.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:31.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:33:31.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:33:31.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:33:31.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:33:31.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:33:31.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:33:31.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:33:31.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:33:31.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:33:31.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:33:31.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:33:31.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:33:31.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:33:31.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:33:31.321 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.321 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.321 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.322 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.322 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:31.322 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.621003244999997, 2024-04-24T14:33:31.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:33:31.322 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.322 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.322 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.322 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.322 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.322 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:33:31.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.322 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:31.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:31.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:33:31.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:31.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:33:31.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:33:31.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:33:31.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:33:31.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:33:31.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:33:31.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:33:31.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:33:31.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:33:31.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:33:31.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.189284169000004, 2024-04-24T14:33:31.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01155111, 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:31.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:31.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:31.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058602+0000", 2024-04-24T14:33:31.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1239999999999999e-06 2024-04-24T14:33:31.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058609+0000", 2024-04-24T14:33:31.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6719999999999998e-06 2024-04-24T14:33:31.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058626+0000", 2024-04-24T14:33:31.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7569999999999999e-05 2024-04-24T14:33:31.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065271+0000", 2024-04-24T14:33:31.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00664469 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065398+0000", 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012694299999999999 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065598+0000", 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000200216 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067596+0000", 2024-04-24T14:33:31.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019976450000000001 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070055+0000", 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024593710000000001 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070109+0000", 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3690999999999998e-05 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070147+0000", 2024-04-24T14:33:31.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7910000000000001e-05 2024-04-24T14:33:31.333 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.333 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.333 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.333 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.333 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:31.333 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.189244833999993, 2024-04-24T14:33:31.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:33:31.333 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.333 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.333 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.333 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.334 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.334 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:33:31.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.334 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:31.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:31.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:33:31.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:31.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:33:31.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:33:31.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:33:31.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:33:31.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:33:31.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:33:31.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:33:31.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:33:31.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:33:31.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:33:31.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:33:31.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:33:31.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:33:31.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:33:31.338 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.338 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.339 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.339 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.339 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:31.339 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.189203331000002, 2024-04-24T14:33:31.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:33:31.339 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.339 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.339 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.339 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.339 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.339 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:33:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:31.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:31.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:33:31.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:33:31.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:33:31.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:33:31.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:33:31.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:33:31.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:33:31.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:33:31.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:33:31.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:33:31.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:33:31.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:33:31.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:33:31.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:33:31.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:33:31.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:33:31.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:33:31.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:33:31.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:33:31.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:33:31.344 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.344 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.345 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.345 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.345 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:31.345 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.189090898000003, 2024-04-24T14:33:31.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:33:31.345 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.345 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.345 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.345 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.345 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.345 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:33:31.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.346 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:31.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:31.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:33:31.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:31.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:33:31.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:33:31.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:33:31.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:33:31.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:33:31.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:33:31.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:33:31.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:33:31.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:33:31.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:33:31.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:33:31.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:33:31.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:33:31.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:33:31.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:33:31.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:33:31.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:33:31.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:33:31.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:33:31.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:33:31.350 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.350 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.350 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.189062598000007, 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:33:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:33:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:33:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:33:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:33:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:33:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:33:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.188974893999998, 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.356 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.357 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.357 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.357 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:33:31.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.357 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:31.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:31.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:33:31.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:31.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:33:31.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:33:31.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:33:31.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:33:31.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:33:31.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:33:31.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:33:31.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:33:31.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.188852092999994, 2024-04-24T14:33:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:33:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:33:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:33:31.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:33:31.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:33:31.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:33:31.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:33:31.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:33:31.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:33:31.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:33:31.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:33:31.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:33:31.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:33:31.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 67.545612160999994, 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:33:31.367 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:33:31.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:33:31.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:33:31.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:33:31.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:33:31.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:33:31.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:33:31.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:33:31.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:31.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:33:31.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:33:31.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:31.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:33:31.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:33:31.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:31.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:33:31.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:33:31.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:33:31.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:33:31.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:33:31.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:33:31.373 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.373 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.373 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.373 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.373 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.373 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:31.374 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:31.375 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:31.379+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:31.475 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:31.478+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:31.484 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:31.484 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:31.484 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:31.484 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:31.494 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:31.576 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:31.578+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:31.594 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:31.595 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:31.595 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:31.595 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:31.595 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:31.604 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:31.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T14:33:31.677 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:31.678+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:31.744 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:31.744 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:31.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:31.745 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:31.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.745 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:0 3.1 e37/18 3:88c1567c:::rbd_trash_purge_schedule:head v 37'1, mlcod=0'0)", 2024-04-24T14:33:31.745 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:31.745 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.543129200999999, 2024-04-24T14:33:31.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001532315, 2024-04-24T14:33:31.745 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:31.745 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:31.745 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:31.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:31.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:31.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208185+0000", 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208203+0000", 2024-04-24T14:33:31.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.855e-05 2024-04-24T14:33:31.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:31.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208239+0000", 2024-04-24T14:33:31.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5299000000000002e-05 2024-04-24T14:33:31.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:31.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208253+0000", 2024-04-24T14:33:31.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3807e-05 2024-04-24T14:33:31.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:31.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208371+0000", 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118921 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208422+0000", 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0917000000000001e-05 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.209621+0000", 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001198333 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.209719+0000", 2024-04-24T14:33:31.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8717999999999999e-05 2024-04-24T14:33:31.749 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.749 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.749 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.749 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:31.749 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:31.749 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:31.749 INFO:tasks.workunit.client.0.smithi173.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-24T14:33:31.754 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:31.799 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:31.802+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:31.827 INFO:tasks.workunit.client.0.smithi173.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-24T14:33:31.854 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:31.854 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:31.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:31.878 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:31.878+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:31.883 INFO:tasks.workunit.client.0.smithi173.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-24T14:33:31.884 INFO:tasks.workunit.client.0.smithi173.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-24T14:33:31.884 INFO:tasks.workunit.client.0.smithi173.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-24T14:33:31.950 INFO:tasks.workunit.client.0.smithi173.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-24T14:33:31.955 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-24T14:33:31.955 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-24T14:33:31.962 INFO:tasks.workunit:Running workunits matching rbd/rbd-nbd.sh on client.0... 2024-04-24T14:33:31.963 INFO:tasks.workunit:Running workunit rbd/rbd-nbd.sh... 2024-04-24T14:33:31.963 DEBUG:teuthology.orchestra.run.smithi173:workunit test rbd/rbd-nbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=330067dbce47655d1f762ef14d546d2fde16a6a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-04-24T14:33:31.964 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:31.964 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:31.964 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:31.964 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:31.964 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:31.973 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:32.015 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:32.018+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:32.024 INFO:tasks.workunit.client.0.smithi173.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-04-24T14:33:32.026 INFO:tasks.workunit.client.0.smithi173.stderr:+ . /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../standalone/ceph-helpers.sh 2024-04-24T14:33:32.026 INFO:tasks.workunit.client.0.smithi173.stderr:++ TIMEOUT=300 2024-04-24T14:33:32.026 INFO:tasks.workunit.client.0.smithi173.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-04-24T14:33:32.026 INFO:tasks.workunit.client.0.smithi173.stderr:++ MAX_TIMEOUT=15 2024-04-24T14:33:32.026 INFO:tasks.workunit.client.0.smithi173.stderr:++ PG_NUM=4 2024-04-24T14:33:32.026 INFO:tasks.workunit.client.0.smithi173.stderr:++ TMPDIR=/tmp 2024-04-24T14:33:32.027 INFO:tasks.workunit.client.0.smithi173.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-04-24T14:33:32.027 INFO:tasks.workunit.client.0.smithi173.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-04-24T14:33:32.027 INFO:tasks.workunit.client.0.smithi173.stderr:++ type xmlstarlet 2024-04-24T14:33:32.027 INFO:tasks.workunit.client.0.smithi173.stderr:++ XMLSTARLET=xmlstarlet 2024-04-24T14:33:32.027 INFO:tasks.workunit.client.0.smithi173.stderr:+++ uname 2024-04-24T14:33:32.028 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' Linux = FreeBSD ']' 2024-04-24T14:33:32.028 INFO:tasks.workunit.client.0.smithi173.stderr:++ SED=sed 2024-04-24T14:33:32.028 INFO:tasks.workunit.client.0.smithi173.stderr:++ AWK=awk 2024-04-24T14:33:32.029 INFO:tasks.workunit.client.0.smithi173.stderr:+++ stty -a 2024-04-24T14:33:32.029 INFO:tasks.workunit.client.0.smithi173.stderr:+++ head -1 2024-04-24T14:33:32.030 INFO:tasks.workunit.client.0.smithi173.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-04-24T14:33:32.044 INFO:tasks.workunit.client.0.smithi173.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-24T14:33:32.045 INFO:tasks.workunit.client.0.smithi173.stderr:++ termwidth= 2024-04-24T14:33:32.045 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-04-24T14:33:32.045 INFO:tasks.workunit.client.0.smithi173.stderr:++ DIFFCOLOPTS='-y ' 2024-04-24T14:33:32.045 INFO:tasks.workunit.client.0.smithi173.stderr:++ KERNCORE=kernel.core_pattern 2024-04-24T14:33:32.045 INFO:tasks.workunit.client.0.smithi173.stderr:++ EXTRA_OPTS= 2024-04-24T14:33:32.059 INFO:tasks.workunit.client.0.smithi173.stderr:++ test '' = TESTS 2024-04-24T14:33:32.060 INFO:tasks.workunit.client.0.smithi173.stderr:+ POOL=rbd 2024-04-24T14:33:32.060 INFO:tasks.workunit.client.0.smithi173.stderr:+ ANOTHER_POOL=new_default_pool16809 2024-04-24T14:33:32.060 INFO:tasks.workunit.client.0.smithi173.stderr:+ NS=ns 2024-04-24T14:33:32.060 INFO:tasks.workunit.client.0.smithi173.stderr:+ IMAGE=testrbdnbd16809 2024-04-24T14:33:32.060 INFO:tasks.workunit.client.0.smithi173.stderr:+ SIZE=64 2024-04-24T14:33:32.060 INFO:tasks.workunit.client.0.smithi173.stderr:+ DATA= 2024-04-24T14:33:32.060 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:33:32.060 INFO:tasks.workunit.client.0.smithi173.stderr:+ setup 2024-04-24T14:33:32.060 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns x 2024-04-24T14:33:32.060 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -e CMakeCache.txt ']' 2024-04-24T14:33:32.060 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo echo test sudo 2024-04-24T14:33:32.060 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:32.061 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:32.061 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:32.061 INFO:tasks.workunit.client.0.smithi173.stderr:++ which echo 2024-04-24T14:33:32.063 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/echo 2024-04-24T14:33:32.063 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:32.063 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/echo test sudo 2024-04-24T14:33:32.068 INFO:tasks.workunit.client.0.smithi173.stdout:test sudo 2024-04-24T14:33:32.069 INFO:tasks.workunit.client.0.smithi173.stderr:+ trap cleanup INT TERM EXIT 2024-04-24T14:33:32.070 INFO:tasks.workunit.client.0.smithi173.stderr:++ mktemp -d 2024-04-24T14:33:32.079 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:32.082+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:32.103 INFO:tasks.workunit.client.0.smithi173.stderr:+ TEMPDIR=/tmp/tmp.waDYLkcsIV 2024-04-24T14:33:32.103 INFO:tasks.workunit.client.0.smithi173.stderr:+ DATA=/tmp/tmp.waDYLkcsIV/data 2024-04-24T14:33:32.103 INFO:tasks.workunit.client.0.smithi173.stderr:+ dd if=/dev/urandom of=/tmp/tmp.waDYLkcsIV/data bs=1M count=64 2024-04-24T14:33:32.117 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:32.117 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:32.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:32.117 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:32.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.117 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:1 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:32.117 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:32.117 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.921521647999999, 2024-04-24T14:33:32.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002159392, 2024-04-24T14:33:32.117 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:32.118 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:32.118 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:32.118 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:32.118 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:32.118 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 1 2024-04-24T14:33:32.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.118 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:32.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:32.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:32.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:32.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201710+0000", 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201731+0000", 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0835999999999998e-05 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201769+0000", 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7472999999999998e-05 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201796+0000", 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6798999999999999e-05 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201923+0000", 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012770699999999999 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203873+0000", 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00194992 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.919955910999999, 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026872300000000001, 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:32.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:32.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:32.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:32.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:32.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:32.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:32.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:32.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203277+0000", 2024-04-24T14:33:32.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203292+0000", 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4277000000000001e-05 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203327+0000", 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5219000000000003e-05 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203350+0000", 2024-04-24T14:33:32.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2807e-05 2024-04-24T14:33:32.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:32.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203437+0000", 2024-04-24T14:33:32.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7070999999999994e-05 2024-04-24T14:33:32.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:32.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203548+0000", 2024-04-24T14:33:32.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000111576 2024-04-24T14:33:32.124 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:32.124 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:32.124 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:32.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:2 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.919842606, 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047788300000000002, 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:32.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:32.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:32.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:32.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203391+0000", 2024-04-24T14:33:32.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:32.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203400+0000", 2024-04-24T14:33:32.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5860000000000005e-06 2024-04-24T14:33:32.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203421+0000", 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1056e-05 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203434+0000", 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3093e-05 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203623+0000", 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000188646 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:32.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203871+0000", 2024-04-24T14:33:32.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00024812199999999998 2024-04-24T14:33:32.128 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:32.128 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:32.128 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:32.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.128 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:32.128 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:32.128 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.919785201, 2024-04-24T14:33:32.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059468100000000001, 2024-04-24T14:33:32.128 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:32.128 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:32.128 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203449+0000", 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:32.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203469+0000", 2024-04-24T14:33:32.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0003000000000001e-05 2024-04-24T14:33:32.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:32.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203486+0000", 2024-04-24T14:33:32.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7309999999999999e-05 2024-04-24T14:33:32.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203501+0000", 2024-04-24T14:33:32.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.49e-05 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203923+0000", 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042195600000000002 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204045+0000", 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012202200000000001 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:3 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.918414273, 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00093837399999999996, 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 3 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:32.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204819+0000", 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:32.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204833+0000", 2024-04-24T14:33:32.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3346999999999999e-05 2024-04-24T14:33:32.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:32.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204866+0000", 2024-04-24T14:33:32.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3302000000000001e-05 2024-04-24T14:33:32.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204903+0000", 2024-04-24T14:33:32.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7576000000000001e-05 2024-04-24T14:33:32.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205040+0000", 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013616 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205231+0000", 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019118999999999999 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205610+0000", 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037947000000000002 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:32.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:32.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205760+0000", 2024-04-24T14:33:32.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000149259 2024-04-24T14:33:32.136 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:32.136 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:32.136 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:32.136 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:32.136 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:32.136 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:32.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:32.179 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:32.182+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:32.247 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:32.247 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:32.247 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:32.247 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:32.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:32.280 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:32.283+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:32.307 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records in 2024-04-24T14:33:32.307 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records out 2024-04-24T14:33:32.308 INFO:tasks.workunit.client.0.smithi173.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.203086 s, 330 MB/s 2024-04-24T14:33:32.308 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd namespace create rbd/ns 2024-04-24T14:33:32.362 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:32.362 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:32.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:32.381 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:32.383+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:32.470 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:36 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'22, mlcod=13'17)", 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.143798122999996, 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043791400000000001, 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331851+0000", 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331864+0000", 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2767e-05 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331881+0000", 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6586e-05 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331894+0000", 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2976000000000001e-05 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332161+0000", 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026765399999999998 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332197+0000", 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5565999999999998e-05 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336158+0000", 2024-04-24T14:33:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039616820000000002 2024-04-24T14:33:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336232+0000", 2024-04-24T14:33:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3151999999999994e-05 2024-04-24T14:33:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:37 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'23, mlcod=13'18)", 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.142911644999998, 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040417389999999999, 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332738+0000", 2024-04-24T14:33:32.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:32.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332750+0000", 2024-04-24T14:33:32.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2432000000000001e-05 2024-04-24T14:33:32.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332768+0000", 2024-04-24T14:33:32.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7314000000000001e-05 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332780+0000", 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2716e-05 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332925+0000", 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144587 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332955+0000", 2024-04-24T14:33:32.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0246000000000001e-05 2024-04-24T14:33:32.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336698+0000", 2024-04-24T14:33:32.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037430559999999998 2024-04-24T14:33:32.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336781+0000", 2024-04-24T14:33:32.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2447999999999994e-05 2024-04-24T14:33:32.478 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.478 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.478 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:39 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'25, mlcod=13'22)", 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.136793322000003, 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038736740000000001, 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338856+0000", 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338995+0000", 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013868100000000001 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339029+0000", 2024-04-24T14:33:32.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4372000000000002e-05 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339050+0000", 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0656999999999999e-05 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339156+0000", 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106177 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339188+0000", 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.243e-05 2024-04-24T14:33:32.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342648+0000", 2024-04-24T14:33:32.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034601689999999999 2024-04-24T14:33:32.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342731+0000", 2024-04-24T14:33:32.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2534999999999995e-05 2024-04-24T14:33:32.482 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.482 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.482 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:67 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'41, mlcod=13'39)", 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.086057038000007, 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031029009999999999, 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:32.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389592+0000", 2024-04-24T14:33:32.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389603+0000", 2024-04-24T14:33:32.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090000000000001e-05 2024-04-24T14:33:32.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389631+0000", 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7772000000000001e-05 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389646+0000", 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.432e-05 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389732+0000", 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6331999999999999e-05 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389771+0000", 2024-04-24T14:33:32.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8751999999999998e-05 2024-04-24T14:33:32.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392585+0000", 2024-04-24T14:33:32.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028147469999999998 2024-04-24T14:33:32.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392696+0000", 2024-04-24T14:33:32.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111121 2024-04-24T14:33:32.486 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.487 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.487 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.487 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T14:33:32.487 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:32.487 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.084914427000001, 2024-04-24T14:33:32.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030475519999999998, 2024-04-24T14:33:32.487 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.487 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390735+0000", 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390804+0000", 2024-04-24T14:33:32.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9488999999999998e-05 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390832+0000", 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7946e-05 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390847+0000", 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4693e-05 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391508+0000", 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066154799999999998 2024-04-24T14:33:32.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391538+0000", 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0029e-05 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393698+0000", 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002160052 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393784+0000", 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5414999999999995e-05 2024-04-24T14:33:32.490 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:72 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'46, mlcod=13'43)", 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.078740933999995, 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036059460000000001, 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:32.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396908+0000", 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396968+0000", 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9412e-05 2024-04-24T14:33:32.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397010+0000", 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2169000000000002e-05 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397022+0000", 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1749999999999999e-05 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397096+0000", 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4701000000000002e-05 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397126+0000", 2024-04-24T14:33:32.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2024-04-24T14:33:32.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400443+0000", 2024-04-24T14:33:32.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033172140000000002 2024-04-24T14:33:32.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400516+0000", 2024-04-24T14:33:32.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2181999999999994e-05 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:73 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'47, mlcod=13'43)", 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.077893529999997, 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055152170000000002, 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:32.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397756+0000", 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397884+0000", 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012842100000000001 2024-04-24T14:33:32.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397923+0000", 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8942000000000001e-05 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397939+0000", 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6073e-05 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398106+0000", 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016737100000000001 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398137+0000", 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0496000000000001e-05 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403191+0000", 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00505391 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403272+0000", 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1483999999999994e-05 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.498 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:74 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'48, mlcod=13'43)", 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.077094504000002, 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048091760000000001, 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398555+0000", 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398570+0000", 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.558e-05 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398610+0000", 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0028999999999999e-05 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398625+0000", 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398701+0000", 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6507000000000005e-05 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398731+0000", 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9356000000000002e-05 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403281+0000", 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004549796 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403365+0000", 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4685000000000007e-05 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.055781578999998, 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038140460000000002, 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419868+0000", 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419882+0000", 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.45e-05 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419919+0000", 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6939e-05 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419961+0000", 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2148999999999999e-05 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420084+0000", 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122958 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420122+0000", 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8334999999999998e-05 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423616+0000", 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003493566 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423683+0000", 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7149000000000004e-05 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.054369824000005, 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030290730000000002, 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421279+0000", 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421352+0000", 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2251999999999999e-05 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421378+0000", 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6006000000000001e-05 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421388+0000", 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0373e-05 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421473+0000", 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4665000000000003e-05 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421512+0000", 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9619000000000001e-05 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424198+0000", 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026854610000000001 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424310+0000", 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011208 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:87 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'57, mlcod=13'51)", 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.050106865000004, 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002974057, 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425542+0000", 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425614+0000", 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1328999999999994e-05 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425664+0000", 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0491999999999997e-05 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425676+0000", 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.134e-05 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425798+0000", 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012221700000000001 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425860+0000", 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1761999999999996e-05 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428368+0000", 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025084 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428518+0000", 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149884 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.849998139999997, 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002919694, 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625652+0000", 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625651+0000", 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625667+0000", 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5973000000000001e-05 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625689+0000", 2024-04-24T14:33:32.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625705+0000", 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5797000000000001e-05 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625805+0000", 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9238000000000001e-05 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.625837+0000", 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2722000000000003e-05 2024-04-24T14:33:32.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628476+0000", 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026391980000000002 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.628572+0000", 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5754000000000005e-05 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'88)", 2024-04-24T14:33:32.518 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.410060262000002, 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031994179999999999, 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065589+0000", 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065602+0000", 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2737e-05 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065620+0000", 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8546000000000002e-05 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065629+0000", 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1600000000000004e-06 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065729+0000", 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9933999999999996e-05 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065792+0000", 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2712000000000005e-05 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068667+0000", 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028752920000000002 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068790+0000", 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012244400000000001 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:137 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'91)", 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.408704525999994, 2024-04-24T14:33:32.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052524399999999997, 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066945+0000", 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067031+0000", 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6228000000000002e-05 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067060+0000", 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8445999999999998e-05 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067069+0000", 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999996e-06 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067172+0000", 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010285099999999999 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067234+0000", 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2068000000000003e-05 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072090+0000", 2024-04-24T14:33:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048560859999999999 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072199+0000", 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108384 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:138 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'96, mlcod=13'91)", 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.407704761999995, 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053159380000000001, 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067945+0000", 2024-04-24T14:33:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067957+0000", 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1963e-05 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067982+0000", 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5570000000000001e-05 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067994+0000", 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2306e-05 2024-04-24T14:33:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068282+0000", 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028799000000000001 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068307+0000", 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4351000000000001e-05 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073142+0000", 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004835326 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073262+0000", 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119695 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:139 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.406221935000005, 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039447479999999997, 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069428+0000", 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069440+0000", 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2047e-05 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069469+0000", 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8852000000000001e-05 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069477+0000", 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069531+0000", 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4126999999999999e-05 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069559+0000", 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7881e-05 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073270+0000", 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037104519999999999 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073373+0000", 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010380899999999999 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:140 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'92)", 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.405396058999997, 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031722510000000001, 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070253+0000", 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070307+0000", 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3909000000000002e-05 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070330+0000", 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2599000000000002e-05 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070337+0000", 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3030000000000002e-06 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070377+0000", 2024-04-24T14:33:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9916000000000002e-05 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070398+0000", 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0579000000000001e-05 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073377+0000", 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002979013 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073427+0000", 2024-04-24T14:33:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0078999999999999e-05 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:143 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'101, mlcod=13'98)", 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.399210779000001, 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030247590000000001, 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076438+0000", 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076533+0000", 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4443000000000004e-05 2024-04-24T14:33:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076563+0000", 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0386999999999999e-05 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076604+0000", 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1012e-05 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076711+0000", 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010637300000000001 2024-04-24T14:33:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076737+0000", 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6190000000000002e-05 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079368+0000", 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026313529999999999 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079465+0000", 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6426000000000001e-05 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:159 1.0 e14/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'113, mlcod=14'109)", 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.765390672999999, 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052053589999999997, 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710259+0000", 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710272+0000", 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.382e-05 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710303+0000", 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0629000000000001e-05 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710322+0000", 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8859999999999999e-05 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710479+0000", 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157076 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710535+0000", 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6289000000000003e-05 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715391+0000", 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048557139999999997 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715465+0000", 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4301000000000006e-05 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:161 1.0 e14/12 1:30306672:devicehealth::main.db.0000000000000000:head v 14'115, mlcod=14'112)", 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.758279870999999, 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039960450000000002, 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717370+0000", 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717499+0000", 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012877400000000001 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:32.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717513+0000", 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4229999999999999e-05 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717521+0000", 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718040+0000", 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051920099999999995 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:32.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718094+0000", 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859000000000001e-05 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721278+0000", 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031838560000000001 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721367+0000", 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9241999999999997e-05 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:32.549 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:32.550 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:32.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:32.551 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:32.554+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:32.651 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:32.658+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:32.673 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-24T14:33:32.683 INFO:tasks.workunit.client.0.smithi173.stderr:+ for ns in '' ${NS} 2024-04-24T14:33:32.683 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd --dest-pool rbd --dest-namespace '' --no-progress import /tmp/tmp.waDYLkcsIV/data testrbdnbd16809 2024-04-24T14:33:32.705 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:32.705 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:32.705 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:32.705 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:32.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:32.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:33:32.762 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:32.767+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:32.810 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:32.810 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:32.810 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:32.810 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:32.810 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:32.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:32.852 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:32.855+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:32.890 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:32.910 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:32.918 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:32.918 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:32.927 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:32.933 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:32.971 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:32.974+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:33.036 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:33.036 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:33.047 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:33.053 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:33.054+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:33.144 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:33.144 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:33.153 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:33.166 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:33.187 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:33.188 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:33.188 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:33.188 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:33.188 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:33.188 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:33.188 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:33.191 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:33.192 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:33.195 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:33.198+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:33.256 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:33.259+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:33.294 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:33.294 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:33.303 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:33.357 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:33.358+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:33.398 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:33.399 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:33.408 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:33.457 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:33.458+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:33.503 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:33.504 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:33.513 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:33.558 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:33.558+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:33.656 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:33.656 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:33.660 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:33.663+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:33.665 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:33.760 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:33.763+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:33.768 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:33.768 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:33.777 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:33.861 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:33.863+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:33.873 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:33.873 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:33.882 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:33.962 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:33.967+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:33.981 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:33.981 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:33.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:34.063 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:34.067+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:34.100 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:34.100 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:34.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:34.164 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:34.170+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:34.211 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:34.211 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:34.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:34.266 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:34.271+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:34.318 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:34.318 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:34.327 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:34.366 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:34.366+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:34.430 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:34.430 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:34.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:34.467 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:34.470+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:34.539 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:34.539 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:34.548 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:34.568 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:34.571+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:34.645 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:34.645 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:34.654 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:34.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T14:33:34.699 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:34.702+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:34.768 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:34.768 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:34.770 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:34.775+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:34.779 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:34.871 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:34.875+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:34.906 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:34.906 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:34.923 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:34.971 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:34.974+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:34.993 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:33:34.994+0000 7fbf67224700 0 -- 172.21.15.173:0/1814218523 >> [v2:172.21.15.90:6816/20309,v1:172.21.15.90:6817/20309] conn(0x7fbf48038790 msgr2=0x7fbf4803ab70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T14:33:35.001 INFO:tasks.workunit.client.0.smithi173.stderr:+ for ns in '' ${NS} 2024-04-24T14:33:35.001 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd --dest-pool rbd --dest-namespace ns --no-progress import /tmp/tmp.waDYLkcsIV/data testrbdnbd16809 2024-04-24T14:33:35.023 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:35.023 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:35.032 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:35.072 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:35.075+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:35.173 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:35.174+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:35.186 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:35.186 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:35.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:33:35.200 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:35.257 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph osd pool create new_default_pool16809 8 2024-04-24T14:33:35.273 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:35.274+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:35.294 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:35.294 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:35.303 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:35.328 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:35.346 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:35.368 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:35.373 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:35.374+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:35.390 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:35.413 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:35.413 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:35.413 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:35.413 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:35.413 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:35.413 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:35.413 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:35.417 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:35.451 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:0 3.1 e37/18 3:88c1567c:::rbd_trash_purge_schedule:head v 37'1, mlcod=0'0)", 2024-04-24T14:33:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.249571961000001, 2024-04-24T14:33:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001532315, 2024-04-24T14:33:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:35.451 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208185+0000", 2024-04-24T14:33:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208203+0000", 2024-04-24T14:33:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.855e-05 2024-04-24T14:33:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208239+0000", 2024-04-24T14:33:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5299000000000002e-05 2024-04-24T14:33:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208253+0000", 2024-04-24T14:33:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3807e-05 2024-04-24T14:33:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208371+0000", 2024-04-24T14:33:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118921 2024-04-24T14:33:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208422+0000", 2024-04-24T14:33:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0917000000000001e-05 2024-04-24T14:33:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.209621+0000", 2024-04-24T14:33:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001198333 2024-04-24T14:33:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.209719+0000", 2024-04-24T14:33:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8717999999999999e-05 2024-04-24T14:33:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:35.455 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.456 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:35.456 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:35.460 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:35.474 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:35.479+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:35.556 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:35.556 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:35.556 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:35.556 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:35.565 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:35.575 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:35.578+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:35.665 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:35.666 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:35.666 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:35.666 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:35.666 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:35.674 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:35.676 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [1] live_osds: [0, 2, 3, 4, 5] 2024-04-24T14:33:35.676 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:33:35.676 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-24T14:33:35.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T14:33:35.703 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-24T14:33:35.723 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:35.726+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:35.776 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:35.778+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:35.806 INFO:tasks.workunit.client.0.smithi173.stderr:pool 'new_default_pool16809' already exists 2024-04-24T14:33:35.815 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd pool init new_default_pool16809 2024-04-24T14:33:35.835 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:35.835 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:35.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:35.835 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:35.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.835 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:1 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:35.835 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:35.835 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.638991056999998, 2024-04-24T14:33:35.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002159392, 2024-04-24T14:33:35.836 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:35.836 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:35.836 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:35.836 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:35.836 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:35.836 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 1 2024-04-24T14:33:35.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.836 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:35.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:35.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:35.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201710+0000", 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201731+0000", 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0835999999999998e-05 2024-04-24T14:33:35.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:35.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201769+0000", 2024-04-24T14:33:35.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7472999999999998e-05 2024-04-24T14:33:35.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:35.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201796+0000", 2024-04-24T14:33:35.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6798999999999999e-05 2024-04-24T14:33:35.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:35.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201923+0000", 2024-04-24T14:33:35.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012770699999999999 2024-04-24T14:33:35.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.839 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:35.839 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203873+0000", 2024-04-24T14:33:35.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00194992 2024-04-24T14:33:35.839 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.839 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:35.839 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.839 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:35.839 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:35.839 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.637425319999998, 2024-04-24T14:33:35.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026872300000000001, 2024-04-24T14:33:35.840 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:35.840 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:35.840 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:35.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:35.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:35.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:35.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:35.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:35.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203277+0000", 2024-04-24T14:33:35.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:35.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:35.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203292+0000", 2024-04-24T14:33:35.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4277000000000001e-05 2024-04-24T14:33:35.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:35.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203327+0000", 2024-04-24T14:33:35.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5219000000000003e-05 2024-04-24T14:33:35.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:35.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203350+0000", 2024-04-24T14:33:35.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2807e-05 2024-04-24T14:33:35.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:35.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203437+0000", 2024-04-24T14:33:35.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7070999999999994e-05 2024-04-24T14:33:35.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:35.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203548+0000", 2024-04-24T14:33:35.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000111576 2024-04-24T14:33:35.843 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.843 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:35.843 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.843 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:2 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:35.843 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:35.843 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.637312014999999, 2024-04-24T14:33:35.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047788300000000002, 2024-04-24T14:33:35.843 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:35.843 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:35.843 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:35.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:35.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:35.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:35.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:35.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:35.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203391+0000", 2024-04-24T14:33:35.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:35.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:35.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203400+0000", 2024-04-24T14:33:35.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5860000000000005e-06 2024-04-24T14:33:35.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:35.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203421+0000", 2024-04-24T14:33:35.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1056e-05 2024-04-24T14:33:35.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:35.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203434+0000", 2024-04-24T14:33:35.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3093e-05 2024-04-24T14:33:35.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:35.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203623+0000", 2024-04-24T14:33:35.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000188646 2024-04-24T14:33:35.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:35.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203871+0000", 2024-04-24T14:33:35.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00024812199999999998 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.637254609999999, 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059468100000000001, 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:35.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203449+0000", 2024-04-24T14:33:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:35.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203469+0000", 2024-04-24T14:33:35.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0003000000000001e-05 2024-04-24T14:33:35.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:35.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203486+0000", 2024-04-24T14:33:35.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7309999999999999e-05 2024-04-24T14:33:35.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:35.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203501+0000", 2024-04-24T14:33:35.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.49e-05 2024-04-24T14:33:35.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:35.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203923+0000", 2024-04-24T14:33:35.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042195600000000002 2024-04-24T14:33:35.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:35.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204045+0000", 2024-04-24T14:33:35.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012202200000000001 2024-04-24T14:33:35.850 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.850 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:35.850 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.851 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:3 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:35.851 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:35.851 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.635883681999999, 2024-04-24T14:33:35.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00093837399999999996, 2024-04-24T14:33:35.851 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:35.851 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:35.851 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:35.851 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:35.851 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:35.851 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 3 2024-04-24T14:33:35.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.851 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:35.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:35.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:35.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:35.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:35.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:35.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204819+0000", 2024-04-24T14:33:35.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:35.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204833+0000", 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3346999999999999e-05 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204866+0000", 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3302000000000001e-05 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204903+0000", 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7576000000000001e-05 2024-04-24T14:33:35.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:35.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205040+0000", 2024-04-24T14:33:35.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013616 2024-04-24T14:33:35.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:35.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205231+0000", 2024-04-24T14:33:35.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019118999999999999 2024-04-24T14:33:35.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:35.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205610+0000", 2024-04-24T14:33:35.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037947000000000002 2024-04-24T14:33:35.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:35.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205760+0000", 2024-04-24T14:33:35.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000149259 2024-04-24T14:33:35.855 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.855 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:35.855 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.855 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e47/46 cost 1048576)", 2024-04-24T14:33:35.855 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.684271+0000", 2024-04-24T14:33:35.855 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.1564337419999999, 2024-04-24T14:33:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00025325, 2024-04-24T14:33:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684271+0000", 2024-04-24T14:33:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684271+0000", 2024-04-24T14:33:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:35.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684270+0000", 2024-04-24T14:33:35.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:35.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:35.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684288+0000", 2024-04-24T14:33:35.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7844000000000001e-05 2024-04-24T14:33:35.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:35.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684307+0000", 2024-04-24T14:33:35.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8759000000000001e-05 2024-04-24T14:33:35.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:35.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684315+0000", 2024-04-24T14:33:35.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9200000000000004e-06 2024-04-24T14:33:35.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:35.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684368+0000", 2024-04-24T14:33:35.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3409000000000003e-05 2024-04-24T14:33:35.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:35.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684524+0000", 2024-04-24T14:33:35.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015631299999999999 2024-04-24T14:33:35.859 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.859 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:35.859 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.859 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:118 3.2 e47/46 3:774f7988:ns::rbd_directory:head v 47'2, mlcod=0'0)", 2024-04-24T14:33:35.859 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.686690+0000", 2024-04-24T14:33:35.859 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.1540146839999998, 2024-04-24T14:33:35.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001318173, 2024-04-24T14:33:35.859 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:35.859 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:35.859 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686690+0000", 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686690+0000", 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686689+0000", 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:35.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686698+0000", 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8529999999999998e-06 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686714+0000", 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5698000000000001e-05 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686721+0000", 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7719999999999997e-06 2024-04-24T14:33:35.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:35.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686786+0000", 2024-04-24T14:33:35.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5333000000000006e-05 2024-04-24T14:33:35.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:35.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686818+0000", 2024-04-24T14:33:35.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1998000000000001e-05 2024-04-24T14:33:35.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:35.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.687962+0000", 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011436980000000001 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.688008+0000", 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6718999999999998e-05 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:119 3.2 e47/46 3:684f7435:::rbd_id.testrbdnbd16809:head v 47'3, mlcod=37'1)", 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.738094+0000", 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.1026112010000002, 2024-04-24T14:33:35.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016595679999999999, 2024-04-24T14:33:35.864 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:35.864 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:35.864 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:35.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:35.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738094+0000", 2024-04-24T14:33:35.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:35.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738094+0000", 2024-04-24T14:33:35.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:35.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738092+0000", 2024-04-24T14:33:35.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:35.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:35.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738116+0000", 2024-04-24T14:33:35.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3883000000000002e-05 2024-04-24T14:33:35.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:35.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738147+0000", 2024-04-24T14:33:35.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1269000000000002e-05 2024-04-24T14:33:35.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:35.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738159+0000", 2024-04-24T14:33:35.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1443e-05 2024-04-24T14:33:35.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:35.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738290+0000", 2024-04-24T14:33:35.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013122399999999999 2024-04-24T14:33:35.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:35.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738361+0000", 2024-04-24T14:33:35.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1062000000000005e-05 2024-04-24T14:33:35.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:35.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.739670+0000", 2024-04-24T14:33:35.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013087540000000001 2024-04-24T14:33:35.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:35.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.739753+0000", 2024-04-24T14:33:35.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3607000000000001e-05 2024-04-24T14:33:35.867 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.868 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:35.868 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.868 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:120 3.2 e47/46 3:5cea7035:::rbd_info:head v 47'4, mlcod=47'2)", 2024-04-24T14:33:35.868 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.745883+0000", 2024-04-24T14:33:35.868 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.0948218889999999, 2024-04-24T14:33:35.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0014160220000000001, 2024-04-24T14:33:35.868 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:35.868 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:35.868 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:35.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:35.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745883+0000", 2024-04-24T14:33:35.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:35.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745883+0000", 2024-04-24T14:33:35.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:35.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:35.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745881+0000", 2024-04-24T14:33:35.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:35.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:35.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745897+0000", 2024-04-24T14:33:35.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5519000000000002e-05 2024-04-24T14:33:35.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:35.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745952+0000", 2024-04-24T14:33:35.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5071999999999997e-05 2024-04-24T14:33:35.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745968+0000", 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6192999999999999e-05 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.746085+0000", 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117187 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.746137+0000", 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2513999999999999e-05 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:35.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.747239+0000", 2024-04-24T14:33:35.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00110176 2024-04-24T14:33:35.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:35.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:35.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:35.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.747299+0000", 2024-04-24T14:33:35.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9703000000000001e-05 2024-04-24T14:33:35.872 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.872 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:35.872 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.872 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:35.872 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:35.872 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:35.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:35.878 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:35.883+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:35.970 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:33:35.970 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:35.970 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 50, 2024-04-24T14:33:35.970 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "6706935e-d09a-472b-b839-321319fa9e54", 2024-04-24T14:33:35.970 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2024-04-24T14:32:09.059925+0000", 2024-04-24T14:33:35.970 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2024-04-24T14:33:35.694794+0000", 2024-04-24T14:33:35.970 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2024-04-24T14:33:00.483606+0000", 2024-04-24T14:33:35.970 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2024-04-24T14:33:30.655356+0000", 2024-04-24T14:33:35.970 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T14:33:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5806080, 2024-04-24T14:33:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2024-04-24T14:33:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "nodeep-scrub", 2024-04-24T14:33:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "noscrub", 2024-04-24T14:33:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2024-04-24T14:33:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2024-04-24T14:33:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2024-04-24T14:33:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2024-04-24T14:33:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 4, 2024-04-24T14:33:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T14:33:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T14:33:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T14:33:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2024-04-24T14:33:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 4, 2024-04-24T14:33:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 6, 2024-04-24T14:33:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2024-04-24T14:33:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "jewel", 2024-04-24T14:33:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "quincy", 2024-04-24T14:33:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2024-04-24T14:33:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2024-04-24T14:33:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2024-04-24T14:33:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:21.233022+0000", 2024-04-24T14:33:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-24T14:33:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-24T14:33:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:33:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:33:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:33:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:33:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:33:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:33:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:33:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:33:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:33:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:33:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2024-04-24T14:33:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2024-04-24T14:33:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2024-04-24T14:33:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2024-04-24T14:33:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2024-04-24T14:33:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:33:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:33:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:33:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:33:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:33:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:33:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:33:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "14", 2024-04-24T14:33:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-24T14:33:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T14:33:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T14:33:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:33:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:33:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-24T14:33:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-24T14:33:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:33:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:33:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:33:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:33:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:33:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:33:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:33:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:33:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:33:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:33:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:33:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:33:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:33:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:33:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:33:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:33:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:33:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:33:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:33:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:33:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:33:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:33:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:33:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:33:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:33:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:33:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:33:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:33:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:33:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:33:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-24T14:33:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2024-04-24T14:33:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2024-04-24T14:33:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:33:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2024-04-24T14:33:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-24T14:33:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2024-04-24T14:33:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:26.609964+0000", 2024-04-24T14:33:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-24T14:33:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T14:33:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:33:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:33:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:33:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:33:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:33:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:33:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:33:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:33:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:33:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:33:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2024-04-24T14:33:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2024-04-24T14:33:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2024-04-24T14:33:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2024-04-24T14:33:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2024-04-24T14:33:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:33:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:33:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:33:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:33:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:33:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:33:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:33:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "49", 2024-04-24T14:33:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "21", 2024-04-24T14:33:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "21", 2024-04-24T14:33:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "21", 2024-04-24T14:33:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:33:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:33:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 3, 2024-04-24T14:33:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 49, 2024-04-24T14:33:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:33:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:33:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:33:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:33:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [ 2024-04-24T14:33:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: 3 2024-04-24T14:33:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:33:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": 3, 2024-04-24T14:33:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": 3, 2024-04-24T14:33:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:33:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:33:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:33:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:33:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:33:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:33:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:33:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:33:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:33:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:33:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:33:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:33:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:33:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:33:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:33:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:33:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:33:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:33:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:33:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:33:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:33:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:33:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:33:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:33:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:33:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-24T14:33:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "cache", 2024-04-24T14:33:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:33.987564+0000", 2024-04-24T14:33:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8201, 2024-04-24T14:33:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-04-24T14:33:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:33:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:33:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:33:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:33:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:33:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:33:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:33:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:33:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:33:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:33:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 4, 2024-04-24T14:33:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 4, 2024-04-24T14:33:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 4, 2024-04-24T14:33:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 4, 2024-04-24T14:33:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 4, 2024-04-24T14:33:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:33:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:33:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:33:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:33:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:33:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:33:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:33:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "49", 2024-04-24T14:33:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "21", 2024-04-24T14:33:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "21", 2024-04-24T14:33:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "21", 2024-04-24T14:33:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:33:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:33:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 3, 2024-04-24T14:33:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 49, 2024-04-24T14:33:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:33:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:33:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:33:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:33:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:33:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": 2, 2024-04-24T14:33:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:33:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:33:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "writeback", 2024-04-24T14:33:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:33:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 250, 2024-04-24T14:33:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:33:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:33:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:33:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:33:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:33:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:33:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:33:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bloom", 2024-04-24T14:33:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "false_positive_probability": 0.050000000000000003, 2024-04-24T14:33:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "target_size": 0, 2024-04-24T14:33:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "seed": 0 2024-04-24T14:33:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 60, 2024-04-24T14:33:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 8, 2024-04-24T14:33:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:33:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [ 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:33:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:33:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:33:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:33:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:33:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:33:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 4, 2024-04-24T14:33:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "new_default_pool16809", 2024-04-24T14:33:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:33:35.603212+0000", 2024-04-24T14:33:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 32769, 2024-04-24T14:33:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,creating", 2024-04-24T14:33:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:33:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:33:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:33:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:33:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:33:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:33:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:33:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:33:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:33:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:33:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2024-04-24T14:33:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2024-04-24T14:33:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2024-04-24T14:33:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2024-04-24T14:33:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2024-04-24T14:33:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:33:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:33:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:33:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:33:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:33:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:33:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:33:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "50", 2024-04-24T14:33:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-24T14:33:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T14:33:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T14:33:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:33:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:33:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-24T14:33:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-24T14:33:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:33:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:33:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:33:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:33:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:33:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:33:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:33:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:33:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:33:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:33:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:33:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:33:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:33:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:33:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:33:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:33:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:33:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:33:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:33:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:33:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:33:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:33:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:33:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:33:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:33:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:33:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:33:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:33:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:33:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:33:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:33:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": {} 2024-04-24T14:33:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-24T14:33:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-24T14:33:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "a2506b4f-3486-4f51-972f-1d9e4b015346", 2024-04-24T14:33:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:33:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:33:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:33:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:33:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:33:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:33:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:33:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 46, 2024-04-24T14:33:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:33:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:33:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:33:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2024-04-24T14:33:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2024-04-24T14:33:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:33:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2024-04-24T14:33:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2024-04-24T14:33:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:33:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2024-04-24T14:33:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2024-04-24T14:33:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:33:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2024-04-24T14:33:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2024-04-24T14:33:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:33:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6817/20309", 2024-04-24T14:33:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6819/20309", 2024-04-24T14:33:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/20309", 2024-04-24T14:33:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6821/20309", 2024-04-24T14:33:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:33:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:33:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:33:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-24T14:33:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "9e17324f-a615-494f-ba0d-7c8aa8dac95c", 2024-04-24T14:33:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 0, 2024-04-24T14:33:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:33:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:33:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:33:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:33:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:33:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:33:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 18, 2024-04-24T14:33:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 32, 2024-04-24T14:33:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:33:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:33:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2024-04-24T14:33:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2024-04-24T14:33:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:33:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2024-04-24T14:33:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2024-04-24T14:33:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:33:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2024-04-24T14:33:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2024-04-24T14:33:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:33:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2024-04-24T14:33:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2024-04-24T14:33:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20308 2024-04-24T14:33:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/20308", 2024-04-24T14:33:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/20308", 2024-04-24T14:33:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/20308", 2024-04-24T14:33:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/20308", 2024-04-24T14:33:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:33:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "exists" 2024-04-24T14:33:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-24T14:33:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7", 2024-04-24T14:33:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:33:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:33:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:33:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:33:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:33:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:33:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:33:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 46, 2024-04-24T14:33:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:33:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:33:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:33:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2024-04-24T14:33:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2024-04-24T14:33:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:33:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2024-04-24T14:33:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2024-04-24T14:33:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:33:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2024-04-24T14:33:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2024-04-24T14:33:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:33:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2024-04-24T14:33:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2024-04-24T14:33:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:33:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6809/20310", 2024-04-24T14:33:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6811/20310", 2024-04-24T14:33:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6815/20310", 2024-04-24T14:33:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6813/20310", 2024-04-24T14:33:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:33:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:33:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:33:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-24T14:33:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "509c16da-22b2-48e8-aa51-c320c8c4fd8b", 2024-04-24T14:33:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:33:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:33:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:33:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:33:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:33:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:33:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:33:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 46, 2024-04-24T14:33:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:33:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:33:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:33:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6800", 2024-04-24T14:33:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6801", 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6802", 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6803", 2024-04-24T14:33:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:33:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6806", 2024-04-24T14:33:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6807", 2024-04-24T14:33:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:33:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6804", 2024-04-24T14:33:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6805", 2024-04-24T14:33:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:33:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6801/20316", 2024-04-24T14:33:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6803/20316", 2024-04-24T14:33:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6807/20316", 2024-04-24T14:33:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6805/20316", 2024-04-24T14:33:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:33:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:33:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:33:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-24T14:33:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1462066b-4e59-46d4-9b85-1f61ee1d56a6", 2024-04-24T14:33:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:33:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:33:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:33:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:33:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:33:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:33:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:33:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 18, 2024-04-24T14:33:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:33:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:33:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:33:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6808", 2024-04-24T14:33:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6809", 2024-04-24T14:33:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:33:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6810", 2024-04-24T14:33:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6811", 2024-04-24T14:33:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:33:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6814", 2024-04-24T14:33:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6815", 2024-04-24T14:33:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:33:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6812", 2024-04-24T14:33:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6813", 2024-04-24T14:33:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:33:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6809/20318", 2024-04-24T14:33:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6811/20318", 2024-04-24T14:33:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6815/20318", 2024-04-24T14:33:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6813/20318", 2024-04-24T14:33:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:33:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:33:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:33:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-24T14:33:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "4dc954a8-5f3c-4826-b9bc-98cf30519382", 2024-04-24T14:33:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:33:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2024-04-24T14:33:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2024-04-24T14:33:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:33:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:33:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:33:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:33:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 32, 2024-04-24T14:33:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:33:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:33:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:33:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6816", 2024-04-24T14:33:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6817", 2024-04-24T14:33:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:33:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6818", 2024-04-24T14:33:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6819", 2024-04-24T14:33:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:33:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6822", 2024-04-24T14:33:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6823", 2024-04-24T14:33:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:33:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:33:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:33:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6820", 2024-04-24T14:33:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:33:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6821", 2024-04-24T14:33:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:33:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6817/20317", 2024-04-24T14:33:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6819/20317", 2024-04-24T14:33:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6823/20317", 2024-04-24T14:33:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6821/20317", 2024-04-24T14:33:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:33:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:33:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:33:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2024-04-24T14:33:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-24T14:33:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:33:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:33:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:33:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:33:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:33:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.792899+0000", 2024-04-24T14:33:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:33:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-24T14:33:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:33:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:33:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:33:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:33:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:33:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.580414+0000", 2024-04-24T14:33:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 32 2024-04-24T14:33:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-24T14:33:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:33:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:33:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:33:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:33:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:33:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.713336+0000", 2024-04-24T14:33:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:33:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-24T14:33:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:33:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:33:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:33:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:33:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:33:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.579638+0000", 2024-04-24T14:33:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:33:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-24T14:33:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:33:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:33:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:33:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:33:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:33:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.570510+0000", 2024-04-24T14:33:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:33:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-24T14:33:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:33:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:33:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:33:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:33:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2024-04-24T14:33:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.736558+0000", 2024-04-24T14:33:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:33:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [], 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [ 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "2.2", 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: 0, 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: 5 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "2.3", 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: 2, 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: 5 2024-04-24T14:33:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.2", 2024-04-24T14:33:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-24T14:33:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: 3, 2024-04-24T14:33:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: 5 2024-04-24T14:33:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2024-04-24T14:33:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2024-04-24T14:33:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2024-04-24T14:33:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2024-04-24T14:33:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2024-04-24T14:33:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-24T14:33:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-24T14:33:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-24T14:33:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-24T14:33:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2024-04-24T14:33:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2024-04-24T14:33:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-24T14:33:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-24T14:33:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-24T14:33:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-24T14:33:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-24T14:33:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-24T14:33:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 2, 2024-04-24T14:33:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-24T14:33:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2024-04-24T14:33:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2024-04-24T14:33:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2024-04-24T14:33:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2024-04-24T14:33:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2024-04-24T14:33:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2024-04-24T14:33:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2024-04-24T14:33:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2024-04-24T14:33:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2024-04-24T14:33:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2024-04-24T14:33:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.040 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:36.041 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:36.041 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:36.043 INFO:tasks.thrashosds.thrasher:j is {'epoch': 50, 'fsid': '6706935e-d09a-472b-b839-321319fa9e54', 'created': '2024-04-24T14:32:09.059925+0000', 'modified': '2024-04-24T14:33:35.694794+0000', 'last_up_change': '2024-04-24T14:33:00.483606+0000', 'last_in_change': '2024-04-24T14:33:30.655356+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T14:32:21.233022+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T14:32:26.609964+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '21', 'last_force_op_resend_prenautilus': '21', 'last_force_op_resend_preluminous': '21', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-04-24T14:32:33.987564+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '21', 'last_force_op_resend_prenautilus': '21', 'last_force_op_resend_preluminous': '21', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 49, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 2, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'new_default_pool16809', 'create_time': '2024-04-24T14:33:35.603212+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': 'a2506b4f-3486-4f51-972f-1d9e4b015346', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 20309}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6819', 'nonce': 20309}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 20309}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6820', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 20309}]}, 'public_addr': '172.21.15.90:6817/20309', 'cluster_addr': '172.21.15.90:6819/20309', 'heartbeat_back_addr': '172.21.15.90:6823/20309', 'heartbeat_front_addr': '172.21.15.90:6821/20309', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9e17324f-a615-494f-ba0d-7c8aa8dac95c', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 32, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 20308}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 20308}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 20308}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 20308}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 20308}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 20308}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 20308}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 20308}]}, 'public_addr': '172.21.15.90:6801/20308', 'cluster_addr': '172.21.15.90:6803/20308', 'heartbeat_back_addr': '172.21.15.90:6807/20308', 'heartbeat_front_addr': '172.21.15.90:6805/20308', 'state': ['exists']}, {'osd': 2, 'uuid': 'f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 20310}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6810', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 20310}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6814', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 20310}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 20310}]}, 'public_addr': '172.21.15.90:6809/20310', 'cluster_addr': '172.21.15.90:6811/20310', 'heartbeat_back_addr': '172.21.15.90:6815/20310', 'heartbeat_front_addr': '172.21.15.90:6813/20310', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '509c16da-22b2-48e8-aa51-c320c8c4fd8b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6800', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6801', 'nonce': 20316}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6802', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6803', 'nonce': 20316}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6806', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6807', 'nonce': 20316}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6804', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6805', 'nonce': 20316}]}, 'public_addr': '172.21.15.103:6801/20316', 'cluster_addr': '172.21.15.103:6803/20316', 'heartbeat_back_addr': '172.21.15.103:6807/20316', 'heartbeat_front_addr': '172.21.15.103:6805/20316', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1462066b-4e59-46d4-9b85-1f61ee1d56a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6808', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6809', 'nonce': 20318}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6810', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6811', 'nonce': 20318}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6814', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6815', 'nonce': 20318}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6812', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6813', 'nonce': 20318}]}, 'public_addr': '172.21.15.103:6809/20318', 'cluster_addr': '172.21.15.103:6811/20318', 'heartbeat_back_addr': '172.21.15.103:6815/20318', 'heartbeat_front_addr': '172.21.15.103:6813/20318', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4dc954a8-5f3c-4826-b9bc-98cf30519382', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6816', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6817', 'nonce': 20317}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6818', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6819', 'nonce': 20317}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6822', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6823', 'nonce': 20317}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6820', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6821', 'nonce': 20317}]}, 'public_addr': '172.21.15.103:6817/20317', 'cluster_addr': '172.21.15.103:6819/20317', 'heartbeat_back_addr': '172.21.15.103:6823/20317', 'heartbeat_front_addr': '172.21.15.103:6821/20317', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.792899+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.580414+0000', 'dead_epoch': 32}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.713336+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.579638+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.570510+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.736558+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [{'pgid': '2.2', 'osds': [0, 5]}, {'pgid': '2.3', 'osds': [2, 5]}, {'pgid': '3.2', 'osds': [3, 5]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 3, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T14:33:36.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:33:36.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:36.045 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:36.043+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:36.142 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:36.146+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:36.144 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:36.144 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:36.144 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:36.144 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:36.144 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:36.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:36.243 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:36.247+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:36.245 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:36.245 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:36.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:36.245 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:36.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.245 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:36 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'22, mlcod=13'17)", 2024-04-24T14:33:36.245 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:36.245 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.916627923999997, 2024-04-24T14:33:36.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043791400000000001, 2024-04-24T14:33:36.245 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.245 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.246 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:36.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:36.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331851+0000", 2024-04-24T14:33:36.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331864+0000", 2024-04-24T14:33:36.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2767e-05 2024-04-24T14:33:36.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331881+0000", 2024-04-24T14:33:36.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6586e-05 2024-04-24T14:33:36.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331894+0000", 2024-04-24T14:33:36.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2976000000000001e-05 2024-04-24T14:33:36.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332161+0000", 2024-04-24T14:33:36.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026765399999999998 2024-04-24T14:33:36.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332197+0000", 2024-04-24T14:33:36.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5565999999999998e-05 2024-04-24T14:33:36.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336158+0000", 2024-04-24T14:33:36.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039616820000000002 2024-04-24T14:33:36.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336232+0000", 2024-04-24T14:33:36.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3151999999999994e-05 2024-04-24T14:33:36.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.250 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.250 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:37 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'23, mlcod=13'18)", 2024-04-24T14:33:36.250 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:36.250 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.915741445999998, 2024-04-24T14:33:36.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040417389999999999, 2024-04-24T14:33:36.251 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.251 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.251 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:36.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:36.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332738+0000", 2024-04-24T14:33:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332750+0000", 2024-04-24T14:33:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2432000000000001e-05 2024-04-24T14:33:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332768+0000", 2024-04-24T14:33:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7314000000000001e-05 2024-04-24T14:33:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332780+0000", 2024-04-24T14:33:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2716e-05 2024-04-24T14:33:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332925+0000", 2024-04-24T14:33:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144587 2024-04-24T14:33:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332955+0000", 2024-04-24T14:33:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0246000000000001e-05 2024-04-24T14:33:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336698+0000", 2024-04-24T14:33:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037430559999999998 2024-04-24T14:33:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336781+0000", 2024-04-24T14:33:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2447999999999994e-05 2024-04-24T14:33:36.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.255 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.255 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:39 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'25, mlcod=13'22)", 2024-04-24T14:33:36.255 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:36.255 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.909623123000003, 2024-04-24T14:33:36.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038736740000000001, 2024-04-24T14:33:36.255 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.255 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.255 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:36.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:36.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338856+0000", 2024-04-24T14:33:36.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338995+0000", 2024-04-24T14:33:36.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013868100000000001 2024-04-24T14:33:36.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339029+0000", 2024-04-24T14:33:36.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4372000000000002e-05 2024-04-24T14:33:36.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339050+0000", 2024-04-24T14:33:36.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0656999999999999e-05 2024-04-24T14:33:36.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339156+0000", 2024-04-24T14:33:36.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106177 2024-04-24T14:33:36.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339188+0000", 2024-04-24T14:33:36.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.243e-05 2024-04-24T14:33:36.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342648+0000", 2024-04-24T14:33:36.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034601689999999999 2024-04-24T14:33:36.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342731+0000", 2024-04-24T14:33:36.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2534999999999995e-05 2024-04-24T14:33:36.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.259 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.260 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:67 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'41, mlcod=13'39)", 2024-04-24T14:33:36.260 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:36.260 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.858886838999993, 2024-04-24T14:33:36.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031029009999999999, 2024-04-24T14:33:36.260 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.260 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.260 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:36.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:36.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389592+0000", 2024-04-24T14:33:36.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389603+0000", 2024-04-24T14:33:36.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090000000000001e-05 2024-04-24T14:33:36.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389631+0000", 2024-04-24T14:33:36.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7772000000000001e-05 2024-04-24T14:33:36.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389646+0000", 2024-04-24T14:33:36.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.432e-05 2024-04-24T14:33:36.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389732+0000", 2024-04-24T14:33:36.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6331999999999999e-05 2024-04-24T14:33:36.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389771+0000", 2024-04-24T14:33:36.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8751999999999998e-05 2024-04-24T14:33:36.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392585+0000", 2024-04-24T14:33:36.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028147469999999998 2024-04-24T14:33:36.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392696+0000", 2024-04-24T14:33:36.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111121 2024-04-24T14:33:36.264 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.264 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.264 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.264 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T14:33:36.264 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:36.265 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.857744228000001, 2024-04-24T14:33:36.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030475519999999998, 2024-04-24T14:33:36.265 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.265 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.265 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:36.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:36.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390735+0000", 2024-04-24T14:33:36.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390804+0000", 2024-04-24T14:33:36.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9488999999999998e-05 2024-04-24T14:33:36.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390832+0000", 2024-04-24T14:33:36.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7946e-05 2024-04-24T14:33:36.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390847+0000", 2024-04-24T14:33:36.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4693e-05 2024-04-24T14:33:36.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391508+0000", 2024-04-24T14:33:36.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066154799999999998 2024-04-24T14:33:36.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391538+0000", 2024-04-24T14:33:36.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0029e-05 2024-04-24T14:33:36.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393698+0000", 2024-04-24T14:33:36.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002160052 2024-04-24T14:33:36.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393784+0000", 2024-04-24T14:33:36.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5414999999999995e-05 2024-04-24T14:33:36.269 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.269 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.269 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.269 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:72 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'46, mlcod=13'43)", 2024-04-24T14:33:36.269 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:36.269 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.851570734999996, 2024-04-24T14:33:36.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036059460000000001, 2024-04-24T14:33:36.269 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.269 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.269 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:36.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:36.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396908+0000", 2024-04-24T14:33:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396968+0000", 2024-04-24T14:33:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9412e-05 2024-04-24T14:33:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397010+0000", 2024-04-24T14:33:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2169000000000002e-05 2024-04-24T14:33:36.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397022+0000", 2024-04-24T14:33:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1749999999999999e-05 2024-04-24T14:33:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397096+0000", 2024-04-24T14:33:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4701000000000002e-05 2024-04-24T14:33:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397126+0000", 2024-04-24T14:33:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2024-04-24T14:33:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400443+0000", 2024-04-24T14:33:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033172140000000002 2024-04-24T14:33:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400516+0000", 2024-04-24T14:33:36.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2181999999999994e-05 2024-04-24T14:33:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:73 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'47, mlcod=13'43)", 2024-04-24T14:33:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.850723330999998, 2024-04-24T14:33:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055152170000000002, 2024-04-24T14:33:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397756+0000", 2024-04-24T14:33:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397884+0000", 2024-04-24T14:33:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012842100000000001 2024-04-24T14:33:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397923+0000", 2024-04-24T14:33:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8942000000000001e-05 2024-04-24T14:33:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397939+0000", 2024-04-24T14:33:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6073e-05 2024-04-24T14:33:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398106+0000", 2024-04-24T14:33:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016737100000000001 2024-04-24T14:33:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398137+0000", 2024-04-24T14:33:36.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0496000000000001e-05 2024-04-24T14:33:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403191+0000", 2024-04-24T14:33:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00505391 2024-04-24T14:33:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403272+0000", 2024-04-24T14:33:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1483999999999994e-05 2024-04-24T14:33:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.278 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:74 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'48, mlcod=13'43)", 2024-04-24T14:33:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.849924305000002, 2024-04-24T14:33:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048091760000000001, 2024-04-24T14:33:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398555+0000", 2024-04-24T14:33:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398570+0000", 2024-04-24T14:33:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.558e-05 2024-04-24T14:33:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398610+0000", 2024-04-24T14:33:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0028999999999999e-05 2024-04-24T14:33:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398625+0000", 2024-04-24T14:33:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-04-24T14:33:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398701+0000", 2024-04-24T14:33:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6507000000000005e-05 2024-04-24T14:33:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398731+0000", 2024-04-24T14:33:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9356000000000002e-05 2024-04-24T14:33:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403281+0000", 2024-04-24T14:33:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004549796 2024-04-24T14:33:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403365+0000", 2024-04-24T14:33:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4685000000000007e-05 2024-04-24T14:33:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-04-24T14:33:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.828611379999998, 2024-04-24T14:33:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038140460000000002, 2024-04-24T14:33:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419868+0000", 2024-04-24T14:33:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419882+0000", 2024-04-24T14:33:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.45e-05 2024-04-24T14:33:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419919+0000", 2024-04-24T14:33:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6939e-05 2024-04-24T14:33:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419961+0000", 2024-04-24T14:33:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2148999999999999e-05 2024-04-24T14:33:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420084+0000", 2024-04-24T14:33:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122958 2024-04-24T14:33:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420122+0000", 2024-04-24T14:33:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8334999999999998e-05 2024-04-24T14:33:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423616+0000", 2024-04-24T14:33:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003493566 2024-04-24T14:33:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423683+0000", 2024-04-24T14:33:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7149000000000004e-05 2024-04-24T14:33:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T14:33:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.827199625000006, 2024-04-24T14:33:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030290730000000002, 2024-04-24T14:33:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421279+0000", 2024-04-24T14:33:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421352+0000", 2024-04-24T14:33:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2251999999999999e-05 2024-04-24T14:33:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421378+0000", 2024-04-24T14:33:36.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6006000000000001e-05 2024-04-24T14:33:36.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421388+0000", 2024-04-24T14:33:36.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0373e-05 2024-04-24T14:33:36.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421473+0000", 2024-04-24T14:33:36.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4665000000000003e-05 2024-04-24T14:33:36.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421512+0000", 2024-04-24T14:33:36.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9619000000000001e-05 2024-04-24T14:33:36.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424198+0000", 2024-04-24T14:33:36.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026854610000000001 2024-04-24T14:33:36.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424310+0000", 2024-04-24T14:33:36.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011208 2024-04-24T14:33:36.293 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.293 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.293 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.293 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:87 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'57, mlcod=13'51)", 2024-04-24T14:33:36.293 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:36.293 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.822936666000004, 2024-04-24T14:33:36.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002974057, 2024-04-24T14:33:36.293 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.293 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.293 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:36.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:36.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425542+0000", 2024-04-24T14:33:36.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425614+0000", 2024-04-24T14:33:36.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1328999999999994e-05 2024-04-24T14:33:36.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425664+0000", 2024-04-24T14:33:36.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0491999999999997e-05 2024-04-24T14:33:36.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425676+0000", 2024-04-24T14:33:36.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.134e-05 2024-04-24T14:33:36.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425798+0000", 2024-04-24T14:33:36.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012221700000000001 2024-04-24T14:33:36.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425860+0000", 2024-04-24T14:33:36.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1761999999999996e-05 2024-04-24T14:33:36.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428368+0000", 2024-04-24T14:33:36.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025084 2024-04-24T14:33:36.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428518+0000", 2024-04-24T14:33:36.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149884 2024-04-24T14:33:36.297 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.298 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.298 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'88)", 2024-04-24T14:33:36.298 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:36.298 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.182890063000002, 2024-04-24T14:33:36.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031994179999999999, 2024-04-24T14:33:36.298 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.299 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.299 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:36.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:36.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065589+0000", 2024-04-24T14:33:36.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065602+0000", 2024-04-24T14:33:36.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2737e-05 2024-04-24T14:33:36.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065620+0000", 2024-04-24T14:33:36.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8546000000000002e-05 2024-04-24T14:33:36.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065629+0000", 2024-04-24T14:33:36.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1600000000000004e-06 2024-04-24T14:33:36.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065729+0000", 2024-04-24T14:33:36.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9933999999999996e-05 2024-04-24T14:33:36.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065792+0000", 2024-04-24T14:33:36.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2712000000000005e-05 2024-04-24T14:33:36.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068667+0000", 2024-04-24T14:33:36.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028752920000000002 2024-04-24T14:33:36.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068790+0000", 2024-04-24T14:33:36.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012244400000000001 2024-04-24T14:33:36.302 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.303 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.303 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:137 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'91)", 2024-04-24T14:33:36.303 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:36.303 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.181534326999994, 2024-04-24T14:33:36.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052524399999999997, 2024-04-24T14:33:36.303 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.303 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.303 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:36.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:36.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066945+0000", 2024-04-24T14:33:36.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067031+0000", 2024-04-24T14:33:36.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6228000000000002e-05 2024-04-24T14:33:36.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067060+0000", 2024-04-24T14:33:36.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8445999999999998e-05 2024-04-24T14:33:36.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067069+0000", 2024-04-24T14:33:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999996e-06 2024-04-24T14:33:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067172+0000", 2024-04-24T14:33:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010285099999999999 2024-04-24T14:33:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067234+0000", 2024-04-24T14:33:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2068000000000003e-05 2024-04-24T14:33:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072090+0000", 2024-04-24T14:33:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048560859999999999 2024-04-24T14:33:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072199+0000", 2024-04-24T14:33:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108384 2024-04-24T14:33:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:138 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'96, mlcod=13'91)", 2024-04-24T14:33:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.180534562999995, 2024-04-24T14:33:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053159380000000001, 2024-04-24T14:33:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067945+0000", 2024-04-24T14:33:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067957+0000", 2024-04-24T14:33:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1963e-05 2024-04-24T14:33:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067982+0000", 2024-04-24T14:33:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5570000000000001e-05 2024-04-24T14:33:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067994+0000", 2024-04-24T14:33:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2306e-05 2024-04-24T14:33:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068282+0000", 2024-04-24T14:33:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028799000000000001 2024-04-24T14:33:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068307+0000", 2024-04-24T14:33:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4351000000000001e-05 2024-04-24T14:33:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073142+0000", 2024-04-24T14:33:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004835326 2024-04-24T14:33:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073262+0000", 2024-04-24T14:33:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119695 2024-04-24T14:33:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:139 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T14:33:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.179051736000005, 2024-04-24T14:33:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039447479999999997, 2024-04-24T14:33:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069428+0000", 2024-04-24T14:33:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069440+0000", 2024-04-24T14:33:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2047e-05 2024-04-24T14:33:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069469+0000", 2024-04-24T14:33:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8852000000000001e-05 2024-04-24T14:33:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069477+0000", 2024-04-24T14:33:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-04-24T14:33:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069531+0000", 2024-04-24T14:33:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4126999999999999e-05 2024-04-24T14:33:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069559+0000", 2024-04-24T14:33:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7881e-05 2024-04-24T14:33:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073270+0000", 2024-04-24T14:33:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037104519999999999 2024-04-24T14:33:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073373+0000", 2024-04-24T14:33:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010380899999999999 2024-04-24T14:33:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:140 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'92)", 2024-04-24T14:33:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.178225859999998, 2024-04-24T14:33:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031722510000000001, 2024-04-24T14:33:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070253+0000", 2024-04-24T14:33:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070307+0000", 2024-04-24T14:33:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3909000000000002e-05 2024-04-24T14:33:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070330+0000", 2024-04-24T14:33:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2599000000000002e-05 2024-04-24T14:33:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070337+0000", 2024-04-24T14:33:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3030000000000002e-06 2024-04-24T14:33:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070377+0000", 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9916000000000002e-05 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070398+0000", 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0579000000000001e-05 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073377+0000", 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002979013 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073427+0000", 2024-04-24T14:33:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0078999999999999e-05 2024-04-24T14:33:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:143 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'101, mlcod=13'98)", 2024-04-24T14:33:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.172040580000001, 2024-04-24T14:33:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030247590000000001, 2024-04-24T14:33:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076438+0000", 2024-04-24T14:33:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076533+0000", 2024-04-24T14:33:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4443000000000004e-05 2024-04-24T14:33:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076563+0000", 2024-04-24T14:33:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0386999999999999e-05 2024-04-24T14:33:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076604+0000", 2024-04-24T14:33:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1012e-05 2024-04-24T14:33:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076711+0000", 2024-04-24T14:33:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010637300000000001 2024-04-24T14:33:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076737+0000", 2024-04-24T14:33:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6190000000000002e-05 2024-04-24T14:33:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079368+0000", 2024-04-24T14:33:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026313529999999999 2024-04-24T14:33:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079465+0000", 2024-04-24T14:33:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6426000000000001e-05 2024-04-24T14:33:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:159 1.0 e14/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'113, mlcod=14'109)", 2024-04-24T14:33:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 72.538220473999999, 2024-04-24T14:33:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052053589999999997, 2024-04-24T14:33:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710259+0000", 2024-04-24T14:33:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710272+0000", 2024-04-24T14:33:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.382e-05 2024-04-24T14:33:36.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710303+0000", 2024-04-24T14:33:36.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0629000000000001e-05 2024-04-24T14:33:36.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710322+0000", 2024-04-24T14:33:36.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8859999999999999e-05 2024-04-24T14:33:36.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710479+0000", 2024-04-24T14:33:36.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157076 2024-04-24T14:33:36.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710535+0000", 2024-04-24T14:33:36.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6289000000000003e-05 2024-04-24T14:33:36.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715391+0000", 2024-04-24T14:33:36.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048557139999999997 2024-04-24T14:33:36.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715465+0000", 2024-04-24T14:33:36.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4301000000000006e-05 2024-04-24T14:33:36.332 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.332 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.332 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.332 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:161 1.0 e14/12 1:30306672:devicehealth::main.db.0000000000000000:head v 14'115, mlcod=14'112)", 2024-04-24T14:33:36.332 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:36.332 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 72.531109671999999, 2024-04-24T14:33:36.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039960450000000002, 2024-04-24T14:33:36.333 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.333 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.333 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:36.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:36.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717370+0000", 2024-04-24T14:33:36.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:36.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717499+0000", 2024-04-24T14:33:36.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012877400000000001 2024-04-24T14:33:36.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717513+0000", 2024-04-24T14:33:36.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4229999999999999e-05 2024-04-24T14:33:36.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717521+0000", 2024-04-24T14:33:36.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2024-04-24T14:33:36.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718040+0000", 2024-04-24T14:33:36.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051920099999999995 2024-04-24T14:33:36.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718094+0000", 2024-04-24T14:33:36.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859000000000001e-05 2024-04-24T14:33:36.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721278+0000", 2024-04-24T14:33:36.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031838560000000001 2024-04-24T14:33:36.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721367+0000", 2024-04-24T14:33:36.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9241999999999997e-05 2024-04-24T14:33:36.337 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.337 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.337 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.337 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.2.11:3 2.4 2:377edd40:::rbd_namespace:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:33:36.337 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:32.361595+0000", 2024-04-24T14:33:36.337 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.8868852280000001, 2024-04-24T14:33:36.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029985379999999998, 2024-04-24T14:33:36.337 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.338 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:36.338 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.361595+0000", 2024-04-24T14:33:36.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.361595+0000", 2024-04-24T14:33:36.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.361594+0000", 2024-04-24T14:33:36.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:36.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.361600+0000", 2024-04-24T14:33:36.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0630000000000002e-06 2024-04-24T14:33:36.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.361610+0000", 2024-04-24T14:33:36.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0144e-05 2024-04-24T14:33:36.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.361619+0000", 2024-04-24T14:33:36.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4190000000000002e-06 2024-04-24T14:33:36.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.363001+0000", 2024-04-24T14:33:36.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013825249999999999 2024-04-24T14:33:36.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:36.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.363140+0000", 2024-04-24T14:33:36.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013897599999999999 2024-04-24T14:33:36.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.364124+0000", 2024-04-24T14:33:36.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00098390700000000001 2024-04-24T14:33:36.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:36.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.364239+0000", 2024-04-24T14:33:36.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011524299999999999 2024-04-24T14:33:36.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.364594+0000", 2024-04-24T14:33:36.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035433399999999999 2024-04-24T14:33:36.342 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.342 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.342 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.342 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.342 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.342 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:36.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:36.345 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:33:36.346 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:33:36.346 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:36.347+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:36.347 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-24T14:33:35.701490+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":1498352,"kb_used_data":3380,"kb_used_omap":8,"kb_used_meta":1494903,"kb_avail":470360848,"statfs":{"total":483183820800,"available":481649508352,"internally_reserved":0,"allocated":3461120,"data_stored":2047532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8520,"internal_metadata":1530781368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.033743"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T14:33:35.694794+0000","last_change":"2024-04-24T14:33:35.694794+0000","last_active":"2024-04-24T14:33:35.694794+0000","last_peered":"2024-04-24T14:33:35.694794+0000","last_clean":"2024-04-24T14:33:35.694794+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T14:33:35.694794+0000","last_undegraded":"2024-04-24T14:33:35.694794+0000","last_fullsized":"2024-04-24T14:33:35.694794+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":189,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T14:33:29.658871+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:33:29.658871+0000","last_peered":"2024-04-24T14:33:29.658871+0000","last_clean":"2024-04-24T14:33:29.658871+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:33:29.658871+0000","last_undegraded":"2024-04-24T14:33:29.658871+0000","last_fullsized":"2024-04-24T14:33:29.658871+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":4,"reported_epoch":45,"state":"peering","last_fresh":"2024-04-24T14:33:30.664453+0000","last_change":"2024-04-24T14:33:30.664836+0000","last_active":"2024-04-24T14:32:27.283594+0000","last_peered":"2024-04-24T14:32:27.283594+0000","last_clean":"2024-04-24T14:32:27.283594+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T14:33:30.664453+0000","last_undegraded":"2024-04-24T14:33:30.664453+0000","last_fullsized":"2024-04-24T14:33:30.664453+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:36:29.050351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"37'1","reported_seq":19,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-04-24T14:33:23.613239+0000","last_change":"2024-04-24T14:33:15.026785+0000","last_active":"2024-04-24T14:33:23.613239+0000","last_peered":"2024-04-24T14:33:23.613239+0000","last_clean":"2024-04-24T14:32:34.325783+0000","last_became_active":"2024-04-24T14:33:01.511527+0000","last_became_peered":"2024-04-24T14:33:01.511527+0000","last_unstale":"2024-04-24T14:33:23.613239+0000","last_undegraded":"2024-04-24T14:33:13.205459+0000","last_fullsized":"2024-04-24T14:33:01.507466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:45:22.561333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T14:33:35.694794+0000","last_change":"2024-04-24T14:33:35.694794+0000","last_active":"2024-04-24T14:33:35.694794+0000","last_peered":"2024-04-24T14:33:35.694794+0000","last_clean":"2024-04-24T14:33:35.694794+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T14:33:35.694794+0000","last_undegraded":"2024-04-24T14:33:35.694794+0000","last_fullsized":"2024-04-24T14:33:35.694794+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":42,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-04-24T14:33:29.658903+0000","last_change":"2024-04-24T14:33:01.508797+0000","last_active":"2024-04-24T14:33:29.658903+0000","last_peered":"2024-04-24T14:33:29.658903+0000","last_clean":"2024-04-24T14:32:59.492028+0000","last_became_active":"2024-04-24T14:33:01.508797+0000","last_became_peered":"2024-04-24T14:33:01.508797+0000","last_unstale":"2024-04-24T14:33:29.658903+0000","last_undegraded":"2024-04-24T14:33:29.658903+0000","last_fullsized":"2024-04-24T14:33:01.506359+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:57:04.752690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"37'1","reported_seq":35,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T14:33:29.663407+0000","last_change":"2024-04-24T14:32:35.360501+0000","last_active":"2024-04-24T14:33:29.663407+0000","last_peered":"2024-04-24T14:33:29.663407+0000","last_clean":"2024-04-24T14:33:29.663407+0000","last_became_active":"2024-04-24T14:32:35.359880+0000","last_became_peered":"2024-04-24T14:32:35.359880+0000","last_unstale":"2024-04-24T14:33:29.663407+0000","last_undegraded":"2024-04-24T14:33:29.663407+0000","last_fullsized":"2024-04-24T14:33:29.663407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:52:26.111730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T14:33:35.694794+0000","last_change":"2024-04-24T14:33:35.694794+0000","last_active":"2024-04-24T14:33:35.694794+0000","last_peered":"2024-04-24T14:33:35.694794+0000","last_clean":"2024-04-24T14:33:35.694794+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T14:33:35.694794+0000","last_undegraded":"2024-04-24T14:33:35.694794+0000","last_fullsized":"2024-04-24T14:33:35.694794+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-04-24T14:33:29.664072+0000","last_change":"2024-04-24T14:33:01.501910+0000","last_active":"2024-04-24T14:33:29.664072+0000","last_peered":"2024-04-24T14:33:29.664072+0000","last_clean":"2024-04-24T14:32:59.485776+0000","last_became_active":"2024-04-24T14:33:01.501910+0000","last_became_peered":"2024-04-24T14:33:01.501910+0000","last_unstale":"2024-04-24T14:33:29.664072+0000","last_undegraded":"2024-04-24T14:33:29.664072+0000","last_fullsized":"2024-04-24T14:33:01.500778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:29:31.384858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-04-24T14:33:29.664103+0000","last_change":"2024-04-24T14:33:01.501944+0000","last_active":"2024-04-24T14:33:29.664103+0000","last_peered":"2024-04-24T14:33:29.664103+0000","last_clean":"2024-04-24T14:32:34.325783+0000","last_became_active":"2024-04-24T14:33:01.501944+0000","last_became_peered":"2024-04-24T14:33:01.501944+0000","last_unstale":"2024-04-24T14:33:29.664103+0000","last_undegraded":"2024-04-24T14:33:29.664103+0000","last_fullsized":"2024-04-24T14:33:01.500778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:07:25.638522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T14:33:35.694794+0000","last_change":"2024-04-24T14:33:35.694794+0000","last_active":"2024-04-24T14:33:35.694794+0000","last_peered":"2024-04-24T14:33:35.694794+0000","last_clean":"2024-04-24T14:33:35.694794+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T14:33:35.694794+0000","last_undegraded":"2024-04-24T14:33:35.694794+0000","last_fullsized":"2024-04-24T14:33:35.694794+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":39,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-04-24T14:33:23.613232+0000","last_change":"2024-04-24T14:33:01.508881+0000","last_active":"2024-04-24T14:33:23.613232+0000","last_peered":"2024-04-24T14:33:23.613232+0000","last_clean":"2024-04-24T14:32:59.834073+0000","last_became_active":"2024-04-24T14:33:01.508881+0000","last_became_peered":"2024-04-24T14:33:01.508881+0000","last_unstale":"2024-04-24T14:33:23.613232+0000","last_undegraded":"2024-04-24T14:33:23.613232+0000","last_fullsized":"2024-04-24T14:33:01.507385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:47:05.753832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T14:33:27.031874+0000","last_change":"2024-04-24T14:32:35.360857+0000","last_active":"2024-04-24T14:33:27.031874+0000","last_peered":"2024-04-24T14:33:27.031874+0000","last_clean":"2024-04-24T14:33:27.031874+0000","last_became_active":"2024-04-24T14:32:35.360300+0000","last_became_peered":"2024-04-24T14:32:35.360300+0000","last_unstale":"2024-04-24T14:33:27.031874+0000","last_undegraded":"2024-04-24T14:33:27.031874+0000","last_fullsized":"2024-04-24T14:33:27.031874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:16:49.472230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T14:33:35.694794+0000","last_change":"2024-04-24T14:33:35.694794+0000","last_active":"2024-04-24T14:33:35.694794+0000","last_peered":"2024-04-24T14:33:35.694794+0000","last_clean":"2024-04-24T14:33:35.694794+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T14:33:35.694794+0000","last_undegraded":"2024-04-24T14:33:35.694794+0000","last_fullsized":"2024-04-24T14:33:35.694794+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-04-24T14:33:29.663548+0000","last_change":"2024-04-24T14:33:01.502985+0000","last_active":"2024-04-24T14:33:29.663548+0000","last_peered":"2024-04-24T14:33:29.663548+0000","last_clean":"2024-04-24T14:32:27.283594+0000","last_became_active":"2024-04-24T14:33:01.502985+0000","last_became_peered":"2024-04-24T14:33:01.502985+0000","last_unstale":"2024-04-24T14:33:29.663548+0000","last_undegraded":"2024-04-24T14:33:29.663548+0000","last_fullsized":"2024-04-24T14:33:01.501533+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:22:56.636491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T14:33:35.694794+0000","last_change":"2024-04-24T14:33:35.694794+0000","last_active":"2024-04-24T14:33:35.694794+0000","last_peered":"2024-04-24T14:33:35.694794+0000","last_clean":"2024-04-24T14:33:35.694794+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T14:33:35.694794+0000","last_undegraded":"2024-04-24T14:33:35.694794+0000","last_fullsized":"2024-04-24T14:33:35.694794+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":36,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T14:33:29.658902+0000","last_change":"2024-04-24T14:32:28.321917+0000","last_active":"2024-04-24T14:33:29.658902+0000","last_peered":"2024-04-24T14:33:29.658902+0000","last_clean":"2024-04-24T14:33:29.658902+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:33:29.658902+0000","last_undegraded":"2024-04-24T14:33:29.658902+0000","last_fullsized":"2024-04-24T14:33:29.658902+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T14:33:35.694794+0000","last_change":"2024-04-24T14:33:35.694794+0000","last_active":"2024-04-24T14:33:35.694794+0000","last_peered":"2024-04-24T14:33:35.694794+0000","last_clean":"2024-04-24T14:33:35.694794+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T14:33:35.694794+0000","last_undegraded":"2024-04-24T14:33:35.694794+0000","last_fullsized":"2024-04-24T14:33:35.694794+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-04-24T14:33:29.658872+0000","last_change":"2024-04-24T14:33:01.511594+0000","last_active":"2024-04-24T14:33:29.658872+0000","last_peered":"2024-04-24T14:33:29.658872+0000","last_clean":"2024-04-24T14:32:27.283594+0000","last_became_active":"2024-04-24T14:33:01.511594+0000","last_became_peered":"2024-04-24T14:33:01.511594+0000","last_unstale":"2024-04-24T14:33:29.658872+0000","last_undegraded":"2024-04-24T14:33:29.658872+0000","last_fullsized":"2024-04-24T14:33:01.506313+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:14:40.168635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T14:33:35.694794+0000","last_change":"2024-04-24T14:33:35.694794+0000","last_active":"2024-04-24T14:33:35.694794+0000","last_peered":"2024-04-24T14:33:35.694794+0000","last_clean":"2024-04-24T14:33:35.694794+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T14:33:35.694794+0000","last_undegraded":"2024-04-24T14:33:35.694794+0000","last_fullsized":"2024-04-24T14:33:35.694794+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":36,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T14:33:29.658860+0000","last_change":"2024-04-24T14:32:28.321025+0000","last_active":"2024-04-24T14:33:29.658860+0000","last_peered":"2024-04-24T14:33:29.658860+0000","last_clean":"2024-04-24T14:33:29.658860+0000","last_became_active":"2024-04-24T14:32:28.320369+0000","last_became_peered":"2024-04-24T14:32:28.320369+0000","last_unstale":"2024-04-24T14:33:29.658860+0000","last_undegraded":"2024-04-24T14:33:29.658860+0000","last_fullsized":"2024-04-24T14:33:29.658860+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:49:07.246787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":6,"acting":6,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640275,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640275,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297836,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074004,"statfs":{"total":96636764160,"available":96331780096,"internally_reserved":0,"allocated":491520,"data_stored":188179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.63300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":2,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.71799999999999997}]},{"osd":3,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":0.85399999999999998}]},{"osd":5,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.96799999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640273,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302512,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94069328,"statfs":{"total":96636764160,"available":96326991872,"internally_reserved":0,"allocated":1085440,"data_stored":778547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.628},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.83399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Apr 24 14:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":0.73099999999999998}]},{"osd":4,"last update":"Wed Apr 24 14:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.66500000000000004}]},{"osd":5,"last update":"Wed Apr 24 14:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.78600000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640275,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297836,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074004,"statfs":{"total":96636764160,"available":96331780096,"internally_reserved":0,"allocated":491520,"data_stored":188179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":0.63500000000000001}]},{"osd":4,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.78000000000000003}]},{"osd":5,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.747},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.71199999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640268,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297656,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074184,"statfs":{"total":96636764160,"available":96331964416,"internally_reserved":0,"allocated":307200,"data_stored":114080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]}]},{"osd":0,"up_from":11,"seq":47244640274,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302512,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94069328,"statfs":{"total":96636764160,"available":96326991872,"internally_reserved":0,"allocated":1085440,"data_stored":778547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":2,"last update":"Wed Apr 24 14:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.876}]},{"osd":3,"last update":"Wed Apr 24 14:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.749}]},{"osd":4,"last update":"Wed Apr 24 14:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.93100000000000005}]},{"osd":5,"last update":"Wed Apr 24 14:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.84699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:33:36.349 INFO:tasks.thrashosds.thrasher:Setting 4.0 to [4, 0, 3, 5] 2024-04-24T14:33:36.349 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.0', '4', '0', '3', '5'] 2024-04-24T14:33:36.349 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 4.0 4 0 3 5 2024-04-24T14:33:36.444 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:36.447+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:36.491 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:36.491 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:36.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:36.546 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:36.546+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:36.594 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:36.594 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:36.594 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:36.594 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:36.594 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:36.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:36.647 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:36.650+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:36.714 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:1 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.514352753000001, 2024-04-24T14:33:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038534000000000002, 2024-04-24T14:33:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206414+0000", 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206427+0000", 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3457e-05 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206450+0000", 2024-04-24T14:33:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2966999999999999e-05 2024-04-24T14:33:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206464+0000", 2024-04-24T14:33:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3706999999999999e-05 2024-04-24T14:33:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206581+0000", 2024-04-24T14:33:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117161 2024-04-24T14:33:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206802+0000", 2024-04-24T14:33:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221108 2024-04-24T14:33:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.514276468999999, 2024-04-24T14:33:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056574800000000003, 2024-04-24T14:33:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206492+0000", 2024-04-24T14:33:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206497+0000", 2024-04-24T14:33:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6570000000000002e-06 2024-04-24T14:33:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206524+0000", 2024-04-24T14:33:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6503e-05 2024-04-24T14:33:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206535+0000", 2024-04-24T14:33:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.092e-05 2024-04-24T14:33:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206859+0000", 2024-04-24T14:33:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032470600000000002 2024-04-24T14:33:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207059+0000", 2024-04-24T14:33:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000199132 2024-04-24T14:33:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:3 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.514174211, 2024-04-24T14:33:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067768899999999998, 2024-04-24T14:33:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206594+0000", 2024-04-24T14:33:36.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:36.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206601+0000", 2024-04-24T14:33:36.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7939999999999997e-06 2024-04-24T14:33:36.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206611+0000", 2024-04-24T14:33:36.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.008e-05 2024-04-24T14:33:36.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206617+0000", 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4629999999999998e-06 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207107+0000", 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048966400000000001 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207273+0000", 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016570199999999999 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.726 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4667.0:1 3.0 3:377edd40:::rbd_namespace:head [call rbd.namespace_add in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:33:36.726 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:32.359801+0000", 2024-04-24T14:33:36.726 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.3609683700000001, 2024-04-24T14:33:36.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039188890000000001, 2024-04-24T14:33:36.726 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.726 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:36.726 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:36.726 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4667", 2024-04-24T14:33:36.726 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:33:36.726 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-24T14:33:36.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.726 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.359801+0000", 2024-04-24T14:33:36.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.359801+0000", 2024-04-24T14:33:36.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.359798+0000", 2024-04-24T14:33:36.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2024-04-24T14:33:36.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.359812+0000", 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3643e-05 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.359834+0000", 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2807e-05 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.359854+0000", 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9355999999999999e-05 2024-04-24T14:33:36.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.359994+0000", 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014065599999999999 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.363715+0000", 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037201769999999999 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.363720+0000", 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2560000000000002e-06 2024-04-24T14:33:36.729 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.730 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.730 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.730 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4863.0:1 3.0 3:3e9fc581:::testrbdnbd16809.rbd:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-24T14:33:36.730 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:32.733719+0000", 2024-04-24T14:33:36.730 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9870502829999999, 2024-04-24T14:33:36.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00120933, 2024-04-24T14:33:36.730 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:36.730 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:36.730 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:36.730 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4863", 2024-04-24T14:33:36.730 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1814218523", 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.733719+0000", 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.733719+0000", 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:36.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.733717+0000", 2024-04-24T14:33:36.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:36.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:36.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.733728+0000", 2024-04-24T14:33:36.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0530000000000001e-05 2024-04-24T14:33:36.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:36.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.733740+0000", 2024-04-24T14:33:36.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2675e-05 2024-04-24T14:33:36.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:36.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.733753+0000", 2024-04-24T14:33:36.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2626000000000001e-05 2024-04-24T14:33:36.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:36.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.733858+0000", 2024-04-24T14:33:36.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104528 2024-04-24T14:33:36.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:36.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:36.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:36.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.734928+0000", 2024-04-24T14:33:36.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010707360000000001 2024-04-24T14:33:36.733 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.734 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.734 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.734 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:36.734 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:36.734 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:36.735 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:36.783 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:36.786+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:36.847 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:36.848 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:36.848 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:36.848 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:36.848 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:36.850+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:36.857 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:36.949 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:36.950+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:36.960 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:36.960 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:36.969 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:37.050 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:37.050+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:37.120 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:37.120 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.796669049000002, 2024-04-24T14:33:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011830553000000001, 2024-04-24T14:33:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 36 2024-04-24T14:33:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:37.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326000+0000", 2024-04-24T14:33:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:37.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326006+0000", 2024-04-24T14:33:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9340000000000003e-06 2024-04-24T14:33:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326044+0000", 2024-04-24T14:33:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8124999999999998e-05 2024-04-24T14:33:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326054+0000", 2024-04-24T14:33:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9650000000000003e-06 2024-04-24T14:33:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332746+0000", 2024-04-24T14:33:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006692539 2024-04-24T14:33:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332833+0000", 2024-04-24T14:33:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6482000000000003e-05 2024-04-24T14:33:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332947+0000", 2024-04-24T14:33:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114472 2024-04-24T14:33:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334320+0000", 2024-04-24T14:33:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001372489 2024-04-24T14:33:37.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337717+0000", 2024-04-24T14:33:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003396849 2024-04-24T14:33:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337789+0000", 2024-04-24T14:33:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2080000000000001e-05 2024-04-24T14:33:37.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337831+0000", 2024-04-24T14:33:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.244e-05 2024-04-24T14:33:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:37.127 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.796607187000006, 2024-04-24T14:33:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:33:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:33:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:33:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:33:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:33:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:33:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:33:37.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:33:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:33:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:33:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:33:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:33:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:33:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:33:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:33:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:33:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:33:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:33:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:33:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:33:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:33:37.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:33:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:33:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:37.134 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.497092245000005, 2024-04-24T14:33:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:33:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:33:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:33:37.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:33:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:33:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:33:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:33:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:33:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:33:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:33:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:33:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:33:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:33:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:33:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:33:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:33:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:33:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:33:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:33:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:33:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:33:37.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:33:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:33:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:37.141 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.496866476999998, 2024-04-24T14:33:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:33:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:33:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:33:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:33:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:33:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:33:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:33:37.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:33:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:33:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:33:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:33:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:33:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:33:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:33:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:33:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:33:37.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:33:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:33:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:33:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:33:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:33:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:33:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:33:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.49668939, 2024-04-24T14:33:37.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:33:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:33:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:37.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:33:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:33:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:33:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:33:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:33:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:33:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:33:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:33:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:33:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:33:37.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:33:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:33:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:33:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:33:37.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:33:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:33:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:33:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:33:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:33:37.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:33:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:33:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:37.155 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.496585476000007, 2024-04-24T14:33:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:33:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:33:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:33:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:33:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:33:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:33:37.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:33:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:33:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:33:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:33:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:33:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:33:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:33:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:33:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:33:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:33:37.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:33:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:33:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:33:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:33:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:33:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:33:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:33:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:37.162 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.496289156000003, 2024-04-24T14:33:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:33:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:33:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:33:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:33:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:33:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:33:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:33:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:33:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:33:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:33:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:33:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:33:37.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:33:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:33:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:33:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:33:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:33:37.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:33:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:33:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:33:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:33:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:33:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:33:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.496197025000001, 2024-04-24T14:33:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:33:37.169 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:33:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:37.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:33:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:33:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:33:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:33:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:33:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:33:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:33:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:33:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:33:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:33:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:33:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:33:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:33:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:33:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:33:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:33:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:33:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:33:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:33:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:33:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:33:37.175 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.496111812999999, 2024-04-24T14:33:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:33:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.176 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:33:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:37.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:33:37.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:37.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:33:37.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:33:37.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:33:37.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:33:37.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:33:37.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:33:37.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:33:37.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:33:37.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:33:37.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:33:37.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:33:37.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:33:37.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:33:37.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:33:37.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:33:37.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:33:37.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:33:37.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:33:37.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:33:37.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:33:37.183 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.183 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.183 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.183 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.183 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:37.183 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.495874932000007, 2024-04-24T14:33:37.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:33:37.183 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.183 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.183 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.184 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.184 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.184 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:33:37.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.184 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.184 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:37.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.184 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:37.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:33:37.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:37.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:33:37.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:33:37.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:33:37.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:33:37.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:33:37.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:33:37.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:33:37.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:33:37.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:33:37.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:33:37.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:33:37.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:33:37.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:33:37.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:33:37.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:33:37.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:33:37.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.189 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:33:37.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:33:37.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.189 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:33:37.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:33:37.189 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.189 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.190 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.190 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.190 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:37.190 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.495793410000005, 2024-04-24T14:33:37.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:33:37.190 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.190 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.190 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.190 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.190 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.191 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:33:37.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.191 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:37.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:37.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:33:37.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:37.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:33:37.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:33:37.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:33:37.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:33:37.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:33:37.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:33:37.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:33:37.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:33:37.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:33:37.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:33:37.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:33:37.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:33:37.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:33:37.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:33:37.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:33:37.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:33:37.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:33:37.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:33:37.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:33:37.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:33:37.196 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.196 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.196 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.197 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.197 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:37.197 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.064074333999997, 2024-04-24T14:33:37.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01155111, 2024-04-24T14:33:37.197 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.197 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.197 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.197 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.197 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.197 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 135 2024-04-24T14:33:37.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.198 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:37.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:37.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058595+0000", 2024-04-24T14:33:37.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:37.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058602+0000", 2024-04-24T14:33:37.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1239999999999999e-06 2024-04-24T14:33:37.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058609+0000", 2024-04-24T14:33:37.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6719999999999998e-06 2024-04-24T14:33:37.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.200 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058626+0000", 2024-04-24T14:33:37.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7569999999999999e-05 2024-04-24T14:33:37.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.200 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065271+0000", 2024-04-24T14:33:37.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00664469 2024-04-24T14:33:37.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.201 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.201 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065398+0000", 2024-04-24T14:33:37.201 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012694299999999999 2024-04-24T14:33:37.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.201 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.201 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.065598+0000", 2024-04-24T14:33:37.201 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000200216 2024-04-24T14:33:37.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.202 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.202 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067596+0000", 2024-04-24T14:33:37.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0019976450000000001 2024-04-24T14:33:37.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.202 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.202 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070055+0000", 2024-04-24T14:33:37.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024593710000000001 2024-04-24T14:33:37.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.202 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070109+0000", 2024-04-24T14:33:37.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3690999999999998e-05 2024-04-24T14:33:37.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070147+0000", 2024-04-24T14:33:37.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7910000000000001e-05 2024-04-24T14:33:37.203 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.203 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.203 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.204 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.204 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:37.204 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.064034999, 2024-04-24T14:33:37.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:33:37.204 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.204 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.204 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.204 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.204 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.204 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:33:37.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.204 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.205 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:37.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.205 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:37.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.206 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.206 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:33:37.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:37.206 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.206 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.206 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.206 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:33:37.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:33:37.206 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.206 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.206 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.206 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:33:37.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:33:37.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:33:37.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:33:37.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:33:37.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:33:37.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:33:37.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:33:37.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:33:37.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:33:37.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:33:37.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:33:37.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:33:37.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:33:37.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:33:37.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:33:37.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.210 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:33:37.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:33:37.210 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.210 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.210 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.211 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.211 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:37.211 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.063993495999995, 2024-04-24T14:33:37.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:33:37.211 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.211 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.211 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.211 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.211 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.211 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:33:37.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.211 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:37.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:37.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:37.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:37.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:33:37.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:33:37.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:33:37.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:33:37.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.214 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:33:37.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:33:37.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.214 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:33:37.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:33:37.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:33:37.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:33:37.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:33:37.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:33:37.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:33:37.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:33:37.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:33:37.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:33:37.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:33:37.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:33:37.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:33:37.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:33:37.217 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.217 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.217 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.217 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.218 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:37.218 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.063881062999997, 2024-04-24T14:33:37.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:33:37.218 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.218 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.218 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.218 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.218 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.218 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:33:37.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.218 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:37.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:37.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:33:37.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:37.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:33:37.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:33:37.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:33:37.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:33:37.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:33:37.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:33:37.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:33:37.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:33:37.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:33:37.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:33:37.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:33:37.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:33:37.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:33:37.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:33:37.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:33:37.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:33:37.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:33:37.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:33:37.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:33:37.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:33:37.224 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.224 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.224 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.224 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.225 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:37.225 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.063852763, 2024-04-24T14:33:37.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:33:37.225 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.225 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.225 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.225 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.225 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.225 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:33:37.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.225 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:37.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:37.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.227 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:33:37.227 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:37.227 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.227 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.227 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.227 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:33:37.227 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:33:37.227 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.227 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.227 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.227 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:33:37.227 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:33:37.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.228 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.228 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:33:37.228 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:33:37.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.228 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.228 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:33:37.228 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:33:37.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:33:37.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:33:37.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:33:37.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:33:37.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:33:37.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:33:37.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.230 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:33:37.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:33:37.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.230 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:33:37.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:33:37.231 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.231 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.231 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.231 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:33:37.231 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:33:37.231 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.231 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.231 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.231 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.231 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.231 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.231 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:37.232 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.063765059000005, 2024-04-24T14:33:37.232 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:33:37.232 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.232 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.232 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.232 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.232 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.232 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:33:37.232 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.232 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.232 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.232 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.233 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:37.233 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.233 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.233 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.233 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:37.233 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.233 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.233 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.233 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:33:37.233 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:37.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:33:37.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:33:37.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:33:37.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:33:37.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.235 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:33:37.235 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:33:37.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.235 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:33:37.235 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:33:37.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.235 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:33:37.236 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:33:37.236 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.236 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.236 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.236 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:33:37.236 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:33:37.236 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.236 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.236 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.236 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:33:37.236 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:33:37.236 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.237 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.237 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.237 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:33:37.237 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:33:37.237 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.237 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.237 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.237 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:33:37.237 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:33:37.237 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.237 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.238 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.238 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:33:37.238 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:33:37.238 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.238 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.238 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.238 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.238 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.238 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:37.238 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:37.238 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.063642258000002, 2024-04-24T14:33:37.239 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:33:37.239 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.239 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.239 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.239 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.239 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.239 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:33:37.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.239 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.239 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.239 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.239 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:37.240 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.240 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.240 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.240 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:37.240 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.240 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.240 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.240 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:33:37.240 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:37.240 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.241 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.241 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:33:37.241 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:33:37.241 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.241 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.241 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.241 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:33:37.241 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:33:37.241 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.241 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.241 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.241 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:33:37.242 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:33:37.242 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.242 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.242 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.242 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:33:37.242 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:33:37.242 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.242 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.242 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.242 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:33:37.242 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:33:37.243 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.243 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.243 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.243 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:33:37.243 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:33:37.243 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.243 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.243 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.243 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:33:37.243 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:33:37.243 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.243 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.243 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.244 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:33:37.244 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:33:37.244 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.244 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.244 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.244 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:33:37.244 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:33:37.244 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.244 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.244 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.244 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:33:37.244 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:33:37.245 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.245 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.245 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.245 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.245 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:33:37.245 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:37.245 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 73.420402326000001, 2024-04-24T14:33:37.245 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:33:37.245 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.245 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.245 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.246 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:37.246 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:37.246 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:33:37.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.246 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.246 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.246 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:37.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:37.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:33:37.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:37.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:33:37.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:33:37.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.248 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:33:37.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:33:37.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.248 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:33:37.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:33:37.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:33:37.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:33:37.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:37.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:33:37.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:33:37.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:37.250 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:33:37.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:33:37.250 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.250 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:37.250 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:33:37.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:33:37.250 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.250 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:37.250 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:33:37.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:33:37.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.251 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:33:37.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:33:37.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.251 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:33:37.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:33:37.251 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.252 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.252 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.252 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.252 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:33:37.252 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:37.252 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.7573976980000001, 2024-04-24T14:33:37.252 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:33:37.252 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:37.252 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:37.252 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:37.252 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:33:37.253 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:33:37.253 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:33:37.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.253 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:37.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.253 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:37.253 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:37.253 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.253 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:37.253 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:37.253 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:37.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:37.254 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:33:37.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:37.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:37.254 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:33:37.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:33:37.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.255 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:37.255 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:33:37.255 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:33:37.255 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.255 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:37.255 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:33:37.255 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:33:37.255 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.255 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.255 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:33:37.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:33:37.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:33:37.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:33:37.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:33:37.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:37.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:33:37.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:33:37.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.257 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:37.257 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:33:37.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:33:37.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:37.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:37.257 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:37.257 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:33:37.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:33:37.257 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.257 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.257 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.258 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:37.258 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:37.258 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:37.259 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:37.260 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:37.262+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:37.360 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:37.363+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:37.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:33:37.420 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:37.420 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:37.420 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:37.420 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:37.430 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:37.460 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:37.463+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:37.527 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:37.528 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:37.528 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:37.528 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:37.528 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:37.537 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:37.560 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:37.563+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:37.564 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:37.583 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:37.606 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:37.629 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:37.643 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:37.643 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:37.652 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:37.652 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:37.652 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:37.652 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:37.652 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:37.652 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:37.652 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:37.653 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:37.657 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:37.698 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:37.703+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:37.706 INFO:teuthology.orchestra.run.smithi090.stderr:set 4.0 pg_upmap_items mapping to [4->0,3->5] 2024-04-24T14:33:37.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T14:33:37.752 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:37.752 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:37.761 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:37.762 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:37.762+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:37.858 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:37.858 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:37.862 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:37.862+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:37.868 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:37.962 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:37.967+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:37.970 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:37.970 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:37.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:38.063 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:38.066+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:38.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:38.164 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:38.170+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:38.180 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:38.181 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:38.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:38.265 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:38.267+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:38.287 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:38.288 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:38.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:38.366 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:38.370+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:38.401 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:38.401 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:38.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:38.467 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:38.470+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:38.506 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:38.507 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:38.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:38.568 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:38.574+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:38.614 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:38.614 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:38.623 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:38.669 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:38.671+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:38.728 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-24T14:33:38.740 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false rbd-nbd 2024-04-24T14:33:38.740 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd-nbd 2024-04-24T14:33:38.743 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:38.743 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:38.753 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:38.770 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:38.770+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:38.783 INFO:tasks.workunit.client.0.smithi173.stderr:rbd-nbd: must specify command 2024-04-24T14:33:38.784 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:38.785 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false rbd-nbd INVALIDCMD 2024-04-24T14:33:38.785 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd-nbd INVALIDCMD 2024-04-24T14:33:38.809 INFO:tasks.workunit.client.0.smithi173.stderr:rbd-nbd: unknown command: INVALIDCMD 2024-04-24T14:33:38.810 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:38.811 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:38.813 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -ne 0 ']' 2024-04-24T14:33:38.813 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false rbd device --device-type nbd map testrbdnbd16809 2024-04-24T14:33:38.813 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd map testrbdnbd16809 2024-04-24T14:33:38.851 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:38.851 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:38.861 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:38.870 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:38.875+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:38.952 INFO:tasks.workunit.client.0.smithi173.stderr:modprobe: ERROR: could not insert 'nbd': Operation not permitted 2024-04-24T14:33:38.955 INFO:tasks.workunit.client.0.smithi173.stderr:rbd-nbd: failed to find unused device 2024-04-24T14:33:38.958 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:38.958 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:38.967 INFO:tasks.workunit.client.0.smithi173.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-04-24T14:33:38.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:38.971 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:38.975+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:38.973 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:38.973 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-04-24T14:33:38.973 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-04-24T14:33:38.973 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:38.974 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:38.975 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:38.976 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:33:38.979 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:33:38.979 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:38.979 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd map INVALIDIMAGE 2024-04-24T14:33:39.064 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:39.064 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:39.071 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:39.075+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:39.073 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:39.095 INFO:tasks.workunit.client.0.smithi173.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-04-24T14:33:39.100 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:39.100 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false _sudo rbd-nbd --device INVALIDDEV map testrbdnbd16809 2024-04-24T14:33:39.100 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd-nbd --device INVALIDDEV map testrbdnbd16809 2024-04-24T14:33:39.100 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:39.101 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:39.102 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:39.103 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd-nbd 2024-04-24T14:33:39.105 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd-nbd 2024-04-24T14:33:39.105 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:39.105 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd-nbd --device INVALIDDEV map testrbdnbd16809 2024-04-24T14:33:39.166 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:39.166 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:39.172 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:39.174+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:39.175 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:39.272 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:39.274+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:39.318 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:39.319 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:39.328 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:39.373 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:39.374+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:39.429 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:39.438 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:39.479 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:39.482+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:39.576 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:39.578+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:39.583 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:39.583 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:39.592 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:39.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:33:39.676 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:39.678+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:39.687 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:39.688 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:39.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:39.742 INFO:tasks.workunit.client.0.smithi173.stderr:rbd-nbd: invalid device path: INVALIDDEV (expected /dev/nbd{num}) 2024-04-24T14:33:39.755 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:39.755 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false rbd device --device-type nbd --format INVALID list 2024-04-24T14:33:39.755 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd --format INVALID list 2024-04-24T14:33:39.776 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:39.778+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:39.780 INFO:tasks.workunit.client.0.smithi173.stderr:rbd: the argument for option '--format' is invalid 2024-04-24T14:33:39.782 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:39.782 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd --format json --pretty-format list 2024-04-24T14:33:39.796 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:39.805 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:39.805 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:39.805 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:39.805 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:39.814 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:39.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:39.837 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:39.841 INFO:tasks.workunit.client.0.smithi173.stdout:[] 2024-04-24T14:33:39.844 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd --format xml list 2024-04-24T14:33:39.858 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:39.878 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:39.878+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:39.879 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:39.879 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:39.879 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:39.879 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:39.879 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:39.879 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:39.879 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:39.883 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:39.901 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:33:39.901 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:33:39.901 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:33:39.902 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:33:39.903 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:33:39.905 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:33:39.905 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:33:39.905 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:33:39.910 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:39.910 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:39.910 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:39.910 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:39.910 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:39.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:39.978 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:39.983+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:40.023 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:40.023 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:40.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:40.023 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:40.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.024 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:36 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'22, mlcod=13'17)", 2024-04-24T14:33:40.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:40.024 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.695489295000002, 2024-04-24T14:33:40.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043791400000000001, 2024-04-24T14:33:40.024 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.024 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.024 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:40.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331853+0000", 2024-04-24T14:33:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331851+0000", 2024-04-24T14:33:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331864+0000", 2024-04-24T14:33:40.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2767e-05 2024-04-24T14:33:40.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331881+0000", 2024-04-24T14:33:40.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6586e-05 2024-04-24T14:33:40.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.331894+0000", 2024-04-24T14:33:40.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2976000000000001e-05 2024-04-24T14:33:40.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332161+0000", 2024-04-24T14:33:40.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026765399999999998 2024-04-24T14:33:40.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332197+0000", 2024-04-24T14:33:40.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5565999999999998e-05 2024-04-24T14:33:40.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336158+0000", 2024-04-24T14:33:40.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039616820000000002 2024-04-24T14:33:40.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336232+0000", 2024-04-24T14:33:40.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3151999999999994e-05 2024-04-24T14:33:40.028 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.028 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.028 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.028 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:37 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'23, mlcod=13'18)", 2024-04-24T14:33:40.028 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:40.029 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.694602817000003, 2024-04-24T14:33:40.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040417389999999999, 2024-04-24T14:33:40.029 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.029 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.029 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:40.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332739+0000", 2024-04-24T14:33:40.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332738+0000", 2024-04-24T14:33:40.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:40.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332750+0000", 2024-04-24T14:33:40.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2432000000000001e-05 2024-04-24T14:33:40.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332768+0000", 2024-04-24T14:33:40.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7314000000000001e-05 2024-04-24T14:33:40.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332780+0000", 2024-04-24T14:33:40.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2716e-05 2024-04-24T14:33:40.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332925+0000", 2024-04-24T14:33:40.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144587 2024-04-24T14:33:40.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.332955+0000", 2024-04-24T14:33:40.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0246000000000001e-05 2024-04-24T14:33:40.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336698+0000", 2024-04-24T14:33:40.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037430559999999998 2024-04-24T14:33:40.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.336781+0000", 2024-04-24T14:33:40.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2447999999999994e-05 2024-04-24T14:33:40.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.033 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.033 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:39 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'25, mlcod=13'22)", 2024-04-24T14:33:40.033 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:40.033 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.688484493999994, 2024-04-24T14:33:40.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038736740000000001, 2024-04-24T14:33:40.033 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.034 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.034 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:40.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338857+0000", 2024-04-24T14:33:40.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338856+0000", 2024-04-24T14:33:40.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.338995+0000", 2024-04-24T14:33:40.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013868100000000001 2024-04-24T14:33:40.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339029+0000", 2024-04-24T14:33:40.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4372000000000002e-05 2024-04-24T14:33:40.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339050+0000", 2024-04-24T14:33:40.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0656999999999999e-05 2024-04-24T14:33:40.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339156+0000", 2024-04-24T14:33:40.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106177 2024-04-24T14:33:40.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.339188+0000", 2024-04-24T14:33:40.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.243e-05 2024-04-24T14:33:40.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342648+0000", 2024-04-24T14:33:40.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034601689999999999 2024-04-24T14:33:40.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.342731+0000", 2024-04-24T14:33:40.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2534999999999995e-05 2024-04-24T14:33:40.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.037 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.038 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.038 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:67 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'41, mlcod=13'39)", 2024-04-24T14:33:40.038 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:40.038 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.637748209999998, 2024-04-24T14:33:40.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031029009999999999, 2024-04-24T14:33:40.038 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.038 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.038 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:40.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389594+0000", 2024-04-24T14:33:40.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389592+0000", 2024-04-24T14:33:40.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389603+0000", 2024-04-24T14:33:40.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1090000000000001e-05 2024-04-24T14:33:40.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389631+0000", 2024-04-24T14:33:40.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7772000000000001e-05 2024-04-24T14:33:40.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389646+0000", 2024-04-24T14:33:40.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.432e-05 2024-04-24T14:33:40.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389732+0000", 2024-04-24T14:33:40.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6331999999999999e-05 2024-04-24T14:33:40.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.389771+0000", 2024-04-24T14:33:40.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8751999999999998e-05 2024-04-24T14:33:40.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392585+0000", 2024-04-24T14:33:40.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028147469999999998 2024-04-24T14:33:40.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.392696+0000", 2024-04-24T14:33:40.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111121 2024-04-24T14:33:40.042 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.042 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.042 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.043 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-04-24T14:33:40.043 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:40.043 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.636605599000006, 2024-04-24T14:33:40.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030475519999999998, 2024-04-24T14:33:40.043 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.043 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.043 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:40.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390736+0000", 2024-04-24T14:33:40.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390735+0000", 2024-04-24T14:33:40.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390804+0000", 2024-04-24T14:33:40.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9488999999999998e-05 2024-04-24T14:33:40.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390832+0000", 2024-04-24T14:33:40.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7946e-05 2024-04-24T14:33:40.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.390847+0000", 2024-04-24T14:33:40.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4693e-05 2024-04-24T14:33:40.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391508+0000", 2024-04-24T14:33:40.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066154799999999998 2024-04-24T14:33:40.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.391538+0000", 2024-04-24T14:33:40.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0029e-05 2024-04-24T14:33:40.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393698+0000", 2024-04-24T14:33:40.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002160052 2024-04-24T14:33:40.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.393784+0000", 2024-04-24T14:33:40.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5414999999999995e-05 2024-04-24T14:33:40.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.047 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.047 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:72 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'46, mlcod=13'43)", 2024-04-24T14:33:40.047 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:40.047 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.630432106000001, 2024-04-24T14:33:40.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036059460000000001, 2024-04-24T14:33:40.048 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.048 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.048 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:40.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396910+0000", 2024-04-24T14:33:40.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396908+0000", 2024-04-24T14:33:40.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.396968+0000", 2024-04-24T14:33:40.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9412e-05 2024-04-24T14:33:40.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397010+0000", 2024-04-24T14:33:40.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2169000000000002e-05 2024-04-24T14:33:40.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397022+0000", 2024-04-24T14:33:40.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1749999999999999e-05 2024-04-24T14:33:40.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397096+0000", 2024-04-24T14:33:40.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4701000000000002e-05 2024-04-24T14:33:40.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397126+0000", 2024-04-24T14:33:40.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0045000000000001e-05 2024-04-24T14:33:40.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400443+0000", 2024-04-24T14:33:40.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033172140000000002 2024-04-24T14:33:40.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.400516+0000", 2024-04-24T14:33:40.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2181999999999994e-05 2024-04-24T14:33:40.052 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.052 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.052 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.052 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:73 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'47, mlcod=13'43)", 2024-04-24T14:33:40.052 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:40.052 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.629584702000002, 2024-04-24T14:33:40.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055152170000000002, 2024-04-24T14:33:40.052 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.052 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.052 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:40.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397757+0000", 2024-04-24T14:33:40.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397756+0000", 2024-04-24T14:33:40.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397884+0000", 2024-04-24T14:33:40.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012842100000000001 2024-04-24T14:33:40.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397923+0000", 2024-04-24T14:33:40.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8942000000000001e-05 2024-04-24T14:33:40.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.397939+0000", 2024-04-24T14:33:40.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6073e-05 2024-04-24T14:33:40.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398106+0000", 2024-04-24T14:33:40.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016737100000000001 2024-04-24T14:33:40.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398137+0000", 2024-04-24T14:33:40.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0496000000000001e-05 2024-04-24T14:33:40.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403191+0000", 2024-04-24T14:33:40.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00505391 2024-04-24T14:33:40.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403272+0000", 2024-04-24T14:33:40.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1483999999999994e-05 2024-04-24T14:33:40.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.056 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.056 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:74 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'48, mlcod=13'43)", 2024-04-24T14:33:40.057 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:40.057 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.628785676000007, 2024-04-24T14:33:40.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048091760000000001, 2024-04-24T14:33:40.057 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.057 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.057 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:40.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398556+0000", 2024-04-24T14:33:40.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398555+0000", 2024-04-24T14:33:40.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398570+0000", 2024-04-24T14:33:40.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.558e-05 2024-04-24T14:33:40.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398610+0000", 2024-04-24T14:33:40.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0028999999999999e-05 2024-04-24T14:33:40.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398625+0000", 2024-04-24T14:33:40.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.485e-05 2024-04-24T14:33:40.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398701+0000", 2024-04-24T14:33:40.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6507000000000005e-05 2024-04-24T14:33:40.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.398731+0000", 2024-04-24T14:33:40.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9356000000000002e-05 2024-04-24T14:33:40.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403281+0000", 2024-04-24T14:33:40.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004549796 2024-04-24T14:33:40.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.403365+0000", 2024-04-24T14:33:40.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4685000000000007e-05 2024-04-24T14:33:40.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.061 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.061 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-04-24T14:33:40.061 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:40.061 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.607472751000003, 2024-04-24T14:33:40.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038140460000000002, 2024-04-24T14:33:40.061 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.061 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.061 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:40.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419869+0000", 2024-04-24T14:33:40.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419868+0000", 2024-04-24T14:33:40.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419882+0000", 2024-04-24T14:33:40.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.45e-05 2024-04-24T14:33:40.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419919+0000", 2024-04-24T14:33:40.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6939e-05 2024-04-24T14:33:40.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.419961+0000", 2024-04-24T14:33:40.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2148999999999999e-05 2024-04-24T14:33:40.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420084+0000", 2024-04-24T14:33:40.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122958 2024-04-24T14:33:40.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.420122+0000", 2024-04-24T14:33:40.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8334999999999998e-05 2024-04-24T14:33:40.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423616+0000", 2024-04-24T14:33:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003493566 2024-04-24T14:33:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.423683+0000", 2024-04-24T14:33:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7149000000000004e-05 2024-04-24T14:33:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.066 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-04-24T14:33:40.066 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:40.066 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.606060995999997, 2024-04-24T14:33:40.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030290730000000002, 2024-04-24T14:33:40.066 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.066 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.066 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:40.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421281+0000", 2024-04-24T14:33:40.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421279+0000", 2024-04-24T14:33:40.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421352+0000", 2024-04-24T14:33:40.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2251999999999999e-05 2024-04-24T14:33:40.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421378+0000", 2024-04-24T14:33:40.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6006000000000001e-05 2024-04-24T14:33:40.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421388+0000", 2024-04-24T14:33:40.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0373e-05 2024-04-24T14:33:40.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421473+0000", 2024-04-24T14:33:40.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4665000000000003e-05 2024-04-24T14:33:40.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.421512+0000", 2024-04-24T14:33:40.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9619000000000001e-05 2024-04-24T14:33:40.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424198+0000", 2024-04-24T14:33:40.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026854610000000001 2024-04-24T14:33:40.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.424310+0000", 2024-04-24T14:33:40.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011208 2024-04-24T14:33:40.070 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.070 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.070 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.070 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:87 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'57, mlcod=13'51)", 2024-04-24T14:33:40.070 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:40.070 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.601798036999995, 2024-04-24T14:33:40.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002974057, 2024-04-24T14:33:40.070 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.071 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.071 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:40.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425544+0000", 2024-04-24T14:33:40.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425542+0000", 2024-04-24T14:33:40.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425614+0000", 2024-04-24T14:33:40.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1328999999999994e-05 2024-04-24T14:33:40.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425664+0000", 2024-04-24T14:33:40.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0491999999999997e-05 2024-04-24T14:33:40.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425676+0000", 2024-04-24T14:33:40.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.134e-05 2024-04-24T14:33:40.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425798+0000", 2024-04-24T14:33:40.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012221700000000001 2024-04-24T14:33:40.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.425860+0000", 2024-04-24T14:33:40.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1761999999999996e-05 2024-04-24T14:33:40.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428368+0000", 2024-04-24T14:33:40.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025084 2024-04-24T14:33:40.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:22.428518+0000", 2024-04-24T14:33:40.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149884 2024-04-24T14:33:40.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.074 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.075 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.075 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'88)", 2024-04-24T14:33:40.075 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:40.075 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.961751434000007, 2024-04-24T14:33:40.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031994179999999999, 2024-04-24T14:33:40.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.075 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:40.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065590+0000", 2024-04-24T14:33:40.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065589+0000", 2024-04-24T14:33:40.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065602+0000", 2024-04-24T14:33:40.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2737e-05 2024-04-24T14:33:40.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065620+0000", 2024-04-24T14:33:40.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8546000000000002e-05 2024-04-24T14:33:40.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065629+0000", 2024-04-24T14:33:40.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1600000000000004e-06 2024-04-24T14:33:40.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065729+0000", 2024-04-24T14:33:40.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9933999999999996e-05 2024-04-24T14:33:40.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.065792+0000", 2024-04-24T14:33:40.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2712000000000005e-05 2024-04-24T14:33:40.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068667+0000", 2024-04-24T14:33:40.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028752920000000002 2024-04-24T14:33:40.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068790+0000", 2024-04-24T14:33:40.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012244400000000001 2024-04-24T14:33:40.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.079 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.079 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:137 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'95, mlcod=13'91)", 2024-04-24T14:33:40.079 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:40.080 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.960395697999999, 2024-04-24T14:33:40.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052524399999999997, 2024-04-24T14:33:40.080 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.080 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.080 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:40.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066946+0000", 2024-04-24T14:33:40.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.066945+0000", 2024-04-24T14:33:40.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067031+0000", 2024-04-24T14:33:40.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6228000000000002e-05 2024-04-24T14:33:40.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067060+0000", 2024-04-24T14:33:40.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8445999999999998e-05 2024-04-24T14:33:40.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067069+0000", 2024-04-24T14:33:40.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5999999999999996e-06 2024-04-24T14:33:40.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067172+0000", 2024-04-24T14:33:40.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010285099999999999 2024-04-24T14:33:40.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067234+0000", 2024-04-24T14:33:40.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2068000000000003e-05 2024-04-24T14:33:40.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072090+0000", 2024-04-24T14:33:40.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048560859999999999 2024-04-24T14:33:40.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.072199+0000", 2024-04-24T14:33:40.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108384 2024-04-24T14:33:40.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.084 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.084 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:138 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'96, mlcod=13'91)", 2024-04-24T14:33:40.084 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:40.084 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.959395934, 2024-04-24T14:33:40.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053159380000000001, 2024-04-24T14:33:40.084 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.084 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.085 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:40.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067946+0000", 2024-04-24T14:33:40.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067945+0000", 2024-04-24T14:33:40.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067957+0000", 2024-04-24T14:33:40.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1963e-05 2024-04-24T14:33:40.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067982+0000", 2024-04-24T14:33:40.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5570000000000001e-05 2024-04-24T14:33:40.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.067994+0000", 2024-04-24T14:33:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2306e-05 2024-04-24T14:33:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068282+0000", 2024-04-24T14:33:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028799000000000001 2024-04-24T14:33:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.068307+0000", 2024-04-24T14:33:40.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4351000000000001e-05 2024-04-24T14:33:40.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073142+0000", 2024-04-24T14:33:40.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004835326 2024-04-24T14:33:40.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073262+0000", 2024-04-24T14:33:40.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119695 2024-04-24T14:33:40.088 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.088 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.089 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.089 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:139 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'97, mlcod=13'92)", 2024-04-24T14:33:40.089 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:40.089 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.957913106999996, 2024-04-24T14:33:40.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039447479999999997, 2024-04-24T14:33:40.089 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.089 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.089 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:40.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069429+0000", 2024-04-24T14:33:40.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069428+0000", 2024-04-24T14:33:40.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:40.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069440+0000", 2024-04-24T14:33:40.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2047e-05 2024-04-24T14:33:40.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069469+0000", 2024-04-24T14:33:40.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8852000000000001e-05 2024-04-24T14:33:40.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069477+0000", 2024-04-24T14:33:40.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6100000000000006e-06 2024-04-24T14:33:40.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069531+0000", 2024-04-24T14:33:40.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4126999999999999e-05 2024-04-24T14:33:40.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.069559+0000", 2024-04-24T14:33:40.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7881e-05 2024-04-24T14:33:40.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073270+0000", 2024-04-24T14:33:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037104519999999999 2024-04-24T14:33:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073373+0000", 2024-04-24T14:33:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010380899999999999 2024-04-24T14:33:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:140 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'98, mlcod=13'92)", 2024-04-24T14:33:40.093 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:40.094 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.957087231000003, 2024-04-24T14:33:40.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031722510000000001, 2024-04-24T14:33:40.094 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.094 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.094 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:40.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070255+0000", 2024-04-24T14:33:40.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070253+0000", 2024-04-24T14:33:40.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:40.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070307+0000", 2024-04-24T14:33:40.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3909000000000002e-05 2024-04-24T14:33:40.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070330+0000", 2024-04-24T14:33:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2599000000000002e-05 2024-04-24T14:33:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070337+0000", 2024-04-24T14:33:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3030000000000002e-06 2024-04-24T14:33:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070377+0000", 2024-04-24T14:33:40.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9916000000000002e-05 2024-04-24T14:33:40.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.070398+0000", 2024-04-24T14:33:40.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0579000000000001e-05 2024-04-24T14:33:40.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073377+0000", 2024-04-24T14:33:40.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002979013 2024-04-24T14:33:40.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.073427+0000", 2024-04-24T14:33:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0078999999999999e-05 2024-04-24T14:33:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:143 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'101, mlcod=13'98)", 2024-04-24T14:33:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.950901951000006, 2024-04-24T14:33:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030247590000000001, 2024-04-24T14:33:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076440+0000", 2024-04-24T14:33:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076438+0000", 2024-04-24T14:33:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076533+0000", 2024-04-24T14:33:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4443000000000004e-05 2024-04-24T14:33:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076563+0000", 2024-04-24T14:33:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0386999999999999e-05 2024-04-24T14:33:40.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076604+0000", 2024-04-24T14:33:40.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1012e-05 2024-04-24T14:33:40.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076711+0000", 2024-04-24T14:33:40.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010637300000000001 2024-04-24T14:33:40.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.076737+0000", 2024-04-24T14:33:40.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6190000000000002e-05 2024-04-24T14:33:40.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079368+0000", 2024-04-24T14:33:40.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026313529999999999 2024-04-24T14:33:40.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.079465+0000", 2024-04-24T14:33:40.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6426000000000001e-05 2024-04-24T14:33:40.102 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.103 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:159 1.0 e14/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'113, mlcod=14'109)", 2024-04-24T14:33:40.103 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:40.103 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.317081845000004, 2024-04-24T14:33:40.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052053589999999997, 2024-04-24T14:33:40.103 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.103 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.103 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:40.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710260+0000", 2024-04-24T14:33:40.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710259+0000", 2024-04-24T14:33:40.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710272+0000", 2024-04-24T14:33:40.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.382e-05 2024-04-24T14:33:40.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710303+0000", 2024-04-24T14:33:40.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0629000000000001e-05 2024-04-24T14:33:40.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710322+0000", 2024-04-24T14:33:40.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8859999999999999e-05 2024-04-24T14:33:40.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710479+0000", 2024-04-24T14:33:40.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000157076 2024-04-24T14:33:40.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.710535+0000", 2024-04-24T14:33:40.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6289000000000003e-05 2024-04-24T14:33:40.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715391+0000", 2024-04-24T14:33:40.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048557139999999997 2024-04-24T14:33:40.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.715465+0000", 2024-04-24T14:33:40.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4301000000000006e-05 2024-04-24T14:33:40.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.107 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.107 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4192.0:161 1.0 e14/12 1:30306672:devicehealth::main.db.0000000000000000:head v 14'115, mlcod=14'112)", 2024-04-24T14:33:40.108 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:40.108 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.309971043000004, 2024-04-24T14:33:40.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039960450000000002, 2024-04-24T14:33:40.108 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.108 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.108 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:40.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717371+0000", 2024-04-24T14:33:40.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717370+0000", 2024-04-24T14:33:40.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:40.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717499+0000", 2024-04-24T14:33:40.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012877400000000001 2024-04-24T14:33:40.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717513+0000", 2024-04-24T14:33:40.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4229999999999999e-05 2024-04-24T14:33:40.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.717521+0000", 2024-04-24T14:33:40.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6960000000000005e-06 2024-04-24T14:33:40.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718040+0000", 2024-04-24T14:33:40.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051920099999999995 2024-04-24T14:33:40.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.718094+0000", 2024-04-24T14:33:40.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859000000000001e-05 2024-04-24T14:33:40.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721278+0000", 2024-04-24T14:33:40.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0031838560000000001 2024-04-24T14:33:40.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:32:23.721367+0000", 2024-04-24T14:33:40.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9241999999999997e-05 2024-04-24T14:33:40.112 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.112 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.112 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.112 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.2.11:3 2.4 2:377edd40:::rbd_namespace:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:33:40.112 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:32.361595+0000", 2024-04-24T14:33:40.112 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.6657465990000002, 2024-04-24T14:33:40.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029985379999999998, 2024-04-24T14:33:40.112 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.112 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:40.113 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.361595+0000", 2024-04-24T14:33:40.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.361595+0000", 2024-04-24T14:33:40.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.361594+0000", 2024-04-24T14:33:40.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:40.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.361600+0000", 2024-04-24T14:33:40.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0630000000000002e-06 2024-04-24T14:33:40.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.361610+0000", 2024-04-24T14:33:40.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0144e-05 2024-04-24T14:33:40.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.361619+0000", 2024-04-24T14:33:40.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4190000000000002e-06 2024-04-24T14:33:40.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.363001+0000", 2024-04-24T14:33:40.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013825249999999999 2024-04-24T14:33:40.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:40.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.363140+0000", 2024-04-24T14:33:40.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013897599999999999 2024-04-24T14:33:40.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.364124+0000", 2024-04-24T14:33:40.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00098390700000000001 2024-04-24T14:33:40.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.364239+0000", 2024-04-24T14:33:40.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011524299999999999 2024-04-24T14:33:40.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.364594+0000", 2024-04-24T14:33:40.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035433399999999999 2024-04-24T14:33:40.117 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.117 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.117 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.117 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.117 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.117 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:40.119 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:33:40.119 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:33:40.119 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:33:40.119 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:33:40.119 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:33:40.119 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:33:40.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:40.120 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:40.118+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:40.130 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=17106 2024-04-24T14:33:40.130 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 17106 2024-04-24T14:33:40.130 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 17106 -C rbd-nbd 2024-04-24T14:33:40.136 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:33:40.137 INFO:tasks.workunit.client.0.smithi173.stdout: 17106 ? 00:00:00 rbd-nbd 2024-04-24T14:33:40.137 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd16809 2024-04-24T14:33:40.137 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd16809 2024-04-24T14:33:40.137 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:40.138 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:40.139 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:40.139 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd-nbd 2024-04-24T14:33:40.141 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd-nbd 2024-04-24T14:33:40.141 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:40.141 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd16809 2024-04-24T14:33:40.220 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:40.222+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:40.227 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:40.227 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:40.227 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:40.227 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:40.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:40.252 INFO:tasks.workunit.client.0.smithi173.stderr:rbd-nbd: the device /dev/nbd0 is busy 2024-04-24T14:33:40.267 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:40.267 INFO:tasks.workunit.client.0.smithi173.stderr:+ dev1=/dev/nbd0 2024-04-24T14:33:40.267 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device /dev/nbd0 17106 2024-04-24T14:33:40.267 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=/dev/nbd0 2024-04-24T14:33:40.267 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=17106 2024-04-24T14:33:40.267 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:33:40.267 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:40.267 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:40.268 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:40.269 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:33:40.271 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:33:40.271 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:40.271 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:33:40.321 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:40.323+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:40.333 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:40.333 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:40.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:40.359 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:33:40.360 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^17106\b' 2024-04-24T14:33:40.360 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^17106\b' 2024-04-24T14:33:40.406 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:40.407 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:33:40.407 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *17106\b' 2024-04-24T14:33:40.407 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *17106\b' 2024-04-24T14:33:40.413 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:40.413 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:33:40.422 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:40.423+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:40.441 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:40.441 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:40.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:40.441 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:40.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.441 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:1 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:40.441 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:40.441 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.240508465000001, 2024-04-24T14:33:40.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038534000000000002, 2024-04-24T14:33:40.442 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.442 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:40.442 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:40.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206417+0000", 2024-04-24T14:33:40.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206414+0000", 2024-04-24T14:33:40.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2024-04-24T14:33:40.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206427+0000", 2024-04-24T14:33:40.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3457e-05 2024-04-24T14:33:40.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206450+0000", 2024-04-24T14:33:40.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2966999999999999e-05 2024-04-24T14:33:40.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206464+0000", 2024-04-24T14:33:40.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3706999999999999e-05 2024-04-24T14:33:40.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206581+0000", 2024-04-24T14:33:40.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117161 2024-04-24T14:33:40.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206802+0000", 2024-04-24T14:33:40.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221108 2024-04-24T14:33:40.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.445 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.445 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:2 2.1 2:88c1567c:::rbd_trash_purge_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:40.445 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:40.445 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.240432180999999, 2024-04-24T14:33:40.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00056574800000000003, 2024-04-24T14:33:40.446 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.446 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:40.446 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:40.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206493+0000", 2024-04-24T14:33:40.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206492+0000", 2024-04-24T14:33:40.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:40.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206497+0000", 2024-04-24T14:33:40.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6570000000000002e-06 2024-04-24T14:33:40.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206524+0000", 2024-04-24T14:33:40.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6503e-05 2024-04-24T14:33:40.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206535+0000", 2024-04-24T14:33:40.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.092e-05 2024-04-24T14:33:40.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206859+0000", 2024-04-24T14:33:40.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032470600000000002 2024-04-24T14:33:40.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207059+0000", 2024-04-24T14:33:40.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000199132 2024-04-24T14:33:40.449 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.449 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.449 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.449 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(osd.0.11:3 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:40.449 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:40.449 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.240329923000001, 2024-04-24T14:33:40.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067768899999999998, 2024-04-24T14:33:40.450 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.450 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:40.450 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:40.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206595+0000", 2024-04-24T14:33:40.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206594+0000", 2024-04-24T14:33:40.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:40.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206601+0000", 2024-04-24T14:33:40.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7939999999999997e-06 2024-04-24T14:33:40.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206611+0000", 2024-04-24T14:33:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.008e-05 2024-04-24T14:33:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.206617+0000", 2024-04-24T14:33:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4629999999999998e-06 2024-04-24T14:33:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207107+0000", 2024-04-24T14:33:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048966400000000001 2024-04-24T14:33:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:13.207273+0000", 2024-04-24T14:33:40.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016570199999999999 2024-04-24T14:33:40.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.453 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.453 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4667.0:1 3.0 3:377edd40:::rbd_namespace:head [call rbd.namespace_add in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:33:40.453 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:32.359801+0000", 2024-04-24T14:33:40.453 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0871240820000008, 2024-04-24T14:33:40.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039188890000000001, 2024-04-24T14:33:40.453 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.454 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:40.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4667", 2024-04-24T14:33:40.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:33:40.454 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-24T14:33:40.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.454 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.359801+0000", 2024-04-24T14:33:40.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.359801+0000", 2024-04-24T14:33:40.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.359798+0000", 2024-04-24T14:33:40.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2024-04-24T14:33:40.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.359812+0000", 2024-04-24T14:33:40.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3643e-05 2024-04-24T14:33:40.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.359834+0000", 2024-04-24T14:33:40.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2807e-05 2024-04-24T14:33:40.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.359854+0000", 2024-04-24T14:33:40.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9355999999999999e-05 2024-04-24T14:33:40.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.359994+0000", 2024-04-24T14:33:40.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014065599999999999 2024-04-24T14:33:40.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:40.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.363715+0000", 2024-04-24T14:33:40.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037201769999999999 2024-04-24T14:33:40.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.363720+0000", 2024-04-24T14:33:40.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2560000000000002e-06 2024-04-24T14:33:40.457 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.457 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.458 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.458 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4863.0:1 3.0 3:3e9fc581:::testrbdnbd16809.rbd:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-24T14:33:40.458 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:32.733719+0000", 2024-04-24T14:33:40.458 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.713205995, 2024-04-24T14:33:40.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00120933, 2024-04-24T14:33:40.458 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.458 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2024-04-24T14:33:40.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:40.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4863", 2024-04-24T14:33:40.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1814218523", 2024-04-24T14:33:40.459 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1 2024-04-24T14:33:40.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.459 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.733719+0000", 2024-04-24T14:33:40.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.733719+0000", 2024-04-24T14:33:40.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.733717+0000", 2024-04-24T14:33:40.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:40.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.733728+0000", 2024-04-24T14:33:40.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0530000000000001e-05 2024-04-24T14:33:40.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.733740+0000", 2024-04-24T14:33:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2675e-05 2024-04-24T14:33:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.733753+0000", 2024-04-24T14:33:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2626000000000001e-05 2024-04-24T14:33:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.733858+0000", 2024-04-24T14:33:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104528 2024-04-24T14:33:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:32.734928+0000", 2024-04-24T14:33:40.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010707360000000001 2024-04-24T14:33:40.462 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.462 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.462 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.462 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4772.0:62 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:40.462 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:40.121967+0000", 2024-04-24T14:33:40.462 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.32495854000000002, 2024-04-24T14:33:40.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00098517200000000009, 2024-04-24T14:33:40.463 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.463 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:40.463 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:40.463 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4772", 2024-04-24T14:33:40.463 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:33:40.463 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 62 2024-04-24T14:33:40.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.463 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.121967+0000", 2024-04-24T14:33:40.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.121967+0000", 2024-04-24T14:33:40.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.121964+0000", 2024-04-24T14:33:40.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2024-04-24T14:33:40.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.121975+0000", 2024-04-24T14:33:40.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1066999999999999e-05 2024-04-24T14:33:40.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.121997+0000", 2024-04-24T14:33:40.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1603e-05 2024-04-24T14:33:40.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.122009+0000", 2024-04-24T14:33:40.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.208e-05 2024-04-24T14:33:40.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.122126+0000", 2024-04-24T14:33:40.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117042 2024-04-24T14:33:40.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.122323+0000", 2024-04-24T14:33:40.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000197345 2024-04-24T14:33:40.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.122952+0000", 2024-04-24T14:33:40.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062866199999999999 2024-04-24T14:33:40.467 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.467 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.467 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.467 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4772.0:133 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:40.467 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:40.301997+0000", 2024-04-24T14:33:40.467 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.14492829199999999, 2024-04-24T14:33:40.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053373400000000001, 2024-04-24T14:33:40.467 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:40.467 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:40.467 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:40.468 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4772", 2024-04-24T14:33:40.468 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:33:40.468 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 133 2024-04-24T14:33:40.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.468 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:40.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:40.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.301997+0000", 2024-04-24T14:33:40.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:40.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.301997+0000", 2024-04-24T14:33:40.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:40.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:40.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.301994+0000", 2024-04-24T14:33:40.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:40.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:40.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.302008+0000", 2024-04-24T14:33:40.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4188999999999999e-05 2024-04-24T14:33:40.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:40.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.302041+0000", 2024-04-24T14:33:40.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2404999999999999e-05 2024-04-24T14:33:40.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.302052+0000", 2024-04-24T14:33:40.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:33:40.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:40.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.302145+0000", 2024-04-24T14:33:40.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2298000000000003e-05 2024-04-24T14:33:40.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:40.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.302305+0000", 2024-04-24T14:33:40.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016080200000000001 2024-04-24T14:33:40.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:40.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:40.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:40.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:40.302530+0000", 2024-04-24T14:33:40.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500299999999999 2024-04-24T14:33:40.471 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.471 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.471 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.471 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:40.472 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:40.472 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:40.473 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:40.522 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:40.522+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:40.579 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:40.579 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:40.580 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:40.580 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:40.589 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:40.623 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:40.626+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:40.692 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:40.692 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:40.692 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:40.692 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:40.692 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:40.701 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:40.724 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:40.727+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:40.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T14:33:40.808 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:40.808 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:40.808 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:40.808 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:40.808 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.808 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.808 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:40.808 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 78.484853510999997, 2024-04-24T14:33:40.809 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011830553000000001, 2024-04-24T14:33:40.809 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.809 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.809 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.809 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.809 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.809 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 36 2024-04-24T14:33:40.809 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.809 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.809 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.809 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.809 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:40.809 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.810 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326001+0000", 2024-04-24T14:33:40.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.810 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326000+0000", 2024-04-24T14:33:40.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:40.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326006+0000", 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9340000000000003e-06 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326044+0000", 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8124999999999998e-05 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326054+0000", 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9650000000000003e-06 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.811 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.812 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332746+0000", 2024-04-24T14:33:40.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006692539 2024-04-24T14:33:40.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.812 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332833+0000", 2024-04-24T14:33:40.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6482000000000003e-05 2024-04-24T14:33:40.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.812 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.332947+0000", 2024-04-24T14:33:40.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114472 2024-04-24T14:33:40.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.813 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.813 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334320+0000", 2024-04-24T14:33:40.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001372489 2024-04-24T14:33:40.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.813 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.813 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337717+0000", 2024-04-24T14:33:40.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003396849 2024-04-24T14:33:40.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.813 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.814 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337789+0000", 2024-04-24T14:33:40.814 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2080000000000001e-05 2024-04-24T14:33:40.814 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.814 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.814 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.814 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.337831+0000", 2024-04-24T14:33:40.814 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.244e-05 2024-04-24T14:33:40.814 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.814 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.814 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.814 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.814 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.814 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.815 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:40.815 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 78.484791649000002, 2024-04-24T14:33:40.815 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:33:40.815 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.815 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.815 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.815 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.815 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.815 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:33:40.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.815 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.815 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.816 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.816 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:40.816 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.816 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.816 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.816 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.816 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:40.816 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.816 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.816 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.816 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.816 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:33:40.816 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:40.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.817 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.817 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:33:40.817 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:33:40.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.817 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.817 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:33:40.817 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:33:40.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.818 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.818 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:33:40.818 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:33:40.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.818 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.818 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.818 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:33:40.818 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:33:40.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.818 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.818 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.818 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:33:40.819 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:33:40.819 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.819 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.819 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.819 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:33:40.819 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:33:40.819 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.819 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.819 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.819 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:33:40.819 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:33:40.819 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.819 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.819 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.820 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.821 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.821 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.821 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.821 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:40.821 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 78.185276707, 2024-04-24T14:33:40.821 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:33:40.821 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.821 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.821 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.821 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.821 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.821 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:33:40.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.822 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.822 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.822 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:40.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.822 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.822 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:40.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:33:40.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:33:40.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:33:40.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:33:40.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:33:40.824 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:33:40.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.825 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.825 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:33:40.825 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:33:40.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.825 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.825 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:33:40.825 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:33:40.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.825 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.826 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:33:40.826 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:33:40.826 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.826 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.826 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.826 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:33:40.826 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:33:40.826 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.826 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.826 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.826 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:33:40.826 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:33:40.827 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.827 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.827 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.827 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.827 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:40.827 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 78.185050939000007, 2024-04-24T14:33:40.827 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:33:40.827 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.827 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.827 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.827 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.828 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.828 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:33:40.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.828 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.828 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.828 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:40.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.828 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.828 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:40.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:33:40.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:33:40.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:33:40.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:33:40.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:33:40.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.830 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.830 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:33:40.830 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:33:40.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.830 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.830 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:33:40.830 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:33:40.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:33:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:33:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:33:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:33:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:33:40.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:33:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:33:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:33:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:33:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:33:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 78.184873851999996, 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.833 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.834 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:33:40.834 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.834 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.834 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.834 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:40.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.834 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.834 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.834 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:40.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.835 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:33:40.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.835 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:33:40.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:33:40.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:33:40.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:33:40.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:33:40.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:33:40.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:33:40.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:33:40.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:33:40.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:33:40.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:33:40.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:33:40.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:33:40.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:33:40.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:33:40.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:33:40.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:33:40.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:33:40.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.839 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.839 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:33:40.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:33:40.839 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.839 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.839 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.839 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.839 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:40.839 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 78.184769938000002, 2024-04-24T14:33:40.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:40.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:33:40.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:40.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:33:40.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:33:40.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:33:40.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:33:40.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:33:40.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:33:40.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:33:40.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:33:40.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:33:40.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:33:40.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:33:40.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:33:40.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:33:40.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:33:40.845 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.845 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.845 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.845 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.845 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:40.845 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 78.184473617999998, 2024-04-24T14:33:40.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:33:40.845 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.845 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.846 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.846 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.846 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.846 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:33:40.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.846 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:40.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:40.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:33:40.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:40.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:33:40.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:33:40.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:33:40.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:33:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:33:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:33:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:33:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:33:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:33:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 78.184381486999996, 2024-04-24T14:33:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:33:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.851 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:33:40.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:33:40.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:33:40.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:33:40.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:33:40.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:33:40.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:33:40.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:33:40.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:33:40.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:33:40.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.856 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.857 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.857 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.857 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:40.857 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 78.184296274999994, 2024-04-24T14:33:40.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:33:40.857 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.857 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.857 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.857 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.857 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.857 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:33:40.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.858 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:40.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:40.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:33:40.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:33:40.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:33:40.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:33:40.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:33:40.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:33:40.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:33:40.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:33:40.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:33:40.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:33:40.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:33:40.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:33:40.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:33:40.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:33:40.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:33:40.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:33:40.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:33:40.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:33:40.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:33:40.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:33:40.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:33:40.862 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.862 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.863 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.863 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.863 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:40.863 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 78.184059394000002, 2024-04-24T14:33:40.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:33:40.863 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.863 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.863 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.863 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.863 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.863 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:33:40.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.864 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:40.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:40.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:33:40.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:33:40.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:33:40.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:33:40.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:33:40.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:33:40.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:33:40.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:33:40.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:33:40.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:33:40.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:33:40.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:33:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:33:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:33:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:33:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:33:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.869 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.869 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:40.869 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 78.183977872, 2024-04-24T14:33:40.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:33:40.869 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.869 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.869 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.869 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.869 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.869 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:33:40.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.869 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:33:40.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:33:40.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:33:40.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:33:40.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:33:40.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:33:40.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:33:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:33:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:33:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:33:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:33:40.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:33:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:33:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:33:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:33:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:33:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.752219460999996, 2024-04-24T14:33:40.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:33:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:33:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:40.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:33:40.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:40.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:33:40.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:33:40.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:33:40.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:33:40.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:33:40.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:33:40.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:33:40.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:33:40.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:33:40.880 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.880 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.880 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.880 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.880 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:40.880 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.752177958000004, 2024-04-24T14:33:40.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:33:40.880 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.880 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.880 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.880 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:33:40.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:33:40.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:33:40.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:33:40.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:33:40.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:33:40.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:33:40.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:33:40.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:33:40.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:33:40.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:33:40.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:33:40.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:33:40.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:33:40.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:33:40.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:33:40.885 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.885 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.885 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.752065525000006, 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:40.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:40.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:33:40.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:40.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:33:40.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:33:40.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:33:40.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:33:40.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:33:40.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:33:40.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:33:40.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:33:40.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:33:40.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:33:40.891 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.891 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.891 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.891 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.891 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:40.891 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.752037224999995, 2024-04-24T14:33:40.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:33:40.892 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.892 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.892 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.892 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.892 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.892 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:33:40.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.892 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:40.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:33:40.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:33:40.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:33:40.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:33:40.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:33:40.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:33:40.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:33:40.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:33:40.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:33:40.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:33:40.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:33:40.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:33:40.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.751949521, 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.897 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.898 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:33:40.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.898 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.898 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.898 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:40.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.898 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.898 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:40.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:33:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:33:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:33:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:33:40.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:33:40.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:33:40.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:33:40.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:33:40.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:33:40.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:33:40.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:33:40.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:33:40.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:33:40.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:33:40.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:33:40.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:33:40.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:33:40.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:33:40.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:33:40.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:33:40.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:33:40.902 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.903 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.903 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.903 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:40.903 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:40.903 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.751826719999997, 2024-04-24T14:33:40.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:33:40.903 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.903 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.903 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.903 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.903 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.903 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:33:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:33:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:33:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:33:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:33:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:33:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:33:40.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:33:40.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:33:40.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:33:40.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:33:40.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:33:40.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:33:40.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:33:40.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:33:40.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:33:40.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.908 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.909 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:33:40.909 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:40.909 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.108586787999997, 2024-04-24T14:33:40.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:33:40.909 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.909 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.909 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.909 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:40.909 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:40.909 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:33:40.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.909 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:40.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:40.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:33:40.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:40.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:33:40.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:33:40.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:33:40.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:33:40.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:33:40.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:33:40.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:33:40.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:33:40.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:33:40.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:33:40.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:40.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:33:40.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:33:40.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:40.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:33:40.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:33:40.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:40.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:33:40.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:33:40.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:33:40.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:33:40.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:33:40.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:33:40.914 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.914 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.914 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.914 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:33:40.914 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.4455821600000007, 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:40.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:33:40.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:40.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:33:40.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:33:40.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:33:40.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:33:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:33:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:33:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.75145816099999996, 2024-04-24T14:33:40.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:40.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:33:40.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:33:40.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:40.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:33:40.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:33:40.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:33:40.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:33:40.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.924 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.925 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.925 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:40.925 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:40.925 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:40.927 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:40.927 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:40.926+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:40.928 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:33:40.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd16809 2024-04-24T14:33:40.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:33:40.928 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:33:40.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:33:40.928 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd-nbd 2024-04-24T14:33:40.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd-nbd 2024-04-24T14:33:40.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:33:40.928 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd16809 2024-04-24T14:33:41.027 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:41.031+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:41.036 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:41.036 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:41.037 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:41.037 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:41.039 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:33:41.039 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' /dev/nbd0 = /dev/nbd0 ']' 2024-04-24T14:33:41.040 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:33:41.040 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep testrbdnbd16809 2024-04-24T14:33:41.046 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:41.112 INFO:tasks.workunit.client.0.smithi173.stdout:17204 rbd testrbdnbd16809 - /dev/nbd0 2024-04-24T14:33:41.112 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:33:41.112 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:33:41.113 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:33:41.113 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:33:41.113 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:33:41.128 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:41.130+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:41.145 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:41.145 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:41.145 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:41.145 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:41.145 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:41.154 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:41.177 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=17204 2024-04-24T14:33:41.177 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 17204 2024-04-24T14:33:41.177 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 17204 -C rbd-nbd 2024-04-24T14:33:41.187 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:33:41.187 INFO:tasks.workunit.client.0.smithi173.stdout: 17204 ? 00:00:00 rbd-nbd 2024-04-24T14:33:41.188 INFO:tasks.workunit.client.0.smithi173.stderr:++ md5sum 2024-04-24T14:33:41.189 INFO:tasks.workunit.client.0.smithi173.stderr:++ dd if=/tmp/tmp.waDYLkcsIV/data bs=1M 2024-04-24T14:33:41.229 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:41.231+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:41.256 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:41.257 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:41.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:41.257 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:41.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.257 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:0 3.1 e37/18 3:88c1567c:::rbd_trash_purge_schedule:head v 37'1, mlcod=0'0)", 2024-04-24T14:33:41.257 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:41.257 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.053939364000001, 2024-04-24T14:33:41.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001532315, 2024-04-24T14:33:41.257 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.257 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:41.257 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.257 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.258 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:41.258 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.258 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.258 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208187+0000", 2024-04-24T14:33:41.258 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.258 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.258 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208185+0000", 2024-04-24T14:33:41.258 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:41.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208203+0000", 2024-04-24T14:33:41.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.855e-05 2024-04-24T14:33:41.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208239+0000", 2024-04-24T14:33:41.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5299000000000002e-05 2024-04-24T14:33:41.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208253+0000", 2024-04-24T14:33:41.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3807e-05 2024-04-24T14:33:41.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.260 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208371+0000", 2024-04-24T14:33:41.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118921 2024-04-24T14:33:41.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:41.260 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.208422+0000", 2024-04-24T14:33:41.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0917000000000001e-05 2024-04-24T14:33:41.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.209621+0000", 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001198333 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.209719+0000", 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8717999999999999e-05 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4703.0:1 4.2 4:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:36.715780+0000", 2024-04-24T14:33:41.261 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.5463465379999999, 2024-04-24T14:33:41.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040425599999999997, 2024-04-24T14:33:41.262 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.262 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:41.262 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:41.262 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4703", 2024-04-24T14:33:41.262 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3712113668", 2024-04-24T14:33:41.262 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 1 2024-04-24T14:33:41.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.262 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.262 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.715780+0000", 2024-04-24T14:33:41.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.263 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.263 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.715780+0000", 2024-04-24T14:33:41.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.263 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.263 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.715778+0000", 2024-04-24T14:33:41.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:41.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.263 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.263 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.715787+0000", 2024-04-24T14:33:41.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4960000000000002e-06 2024-04-24T14:33:41.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.715806+0000", 2024-04-24T14:33:41.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9134999999999999e-05 2024-04-24T14:33:41.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.715815+0000", 2024-04-24T14:33:41.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.354e-06 2024-04-24T14:33:41.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.715912+0000", 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7539000000000002e-05 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.716184+0000", 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00027202200000000002 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4703.0:2 4.2 4:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:36.717017+0000", 2024-04-24T14:33:41.265 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.5451095849999996, 2024-04-24T14:33:41.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000248478, 2024-04-24T14:33:41.266 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.266 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:41.266 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:41.266 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4703", 2024-04-24T14:33:41.266 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3712113668", 2024-04-24T14:33:41.266 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:33:41.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.266 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.266 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.266 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.717017+0000", 2024-04-24T14:33:41.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.267 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.267 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.267 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.267 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.717017+0000", 2024-04-24T14:33:41.267 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.267 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.267 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.267 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.267 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.717016+0000", 2024-04-24T14:33:41.267 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:41.267 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.267 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.267 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.717025+0000", 2024-04-24T14:33:41.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7639999999999994e-06 2024-04-24T14:33:41.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.717040+0000", 2024-04-24T14:33:41.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5191000000000001e-05 2024-04-24T14:33:41.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.717047+0000", 2024-04-24T14:33:41.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7100000000000007e-06 2024-04-24T14:33:41.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.717121+0000", 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.326e-05 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:36.717265+0000", 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014470999999999999 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4772.0:12 3.1 e53/18 3:990e5750:::rbd_header.12ff64619e2d:head v 53'17, mlcod=53'15)", 2024-04-24T14:33:41.269 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.015832+0000", 2024-04-24T14:33:41.270 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.2462946269999999, 2024-04-24T14:33:41.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001381904, 2024-04-24T14:33:41.270 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.270 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:41.270 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.015832+0000", 2024-04-24T14:33:41.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.015832+0000", 2024-04-24T14:33:41.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.015830+0000", 2024-04-24T14:33:41.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:41.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.015849+0000", 2024-04-24T14:33:41.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9454e-05 2024-04-24T14:33:41.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.015879+0000", 2024-04-24T14:33:41.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0321999999999999e-05 2024-04-24T14:33:41.272 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.272 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.272 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.015893+0000", 2024-04-24T14:33:41.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4027e-05 2024-04-24T14:33:41.272 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.272 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.272 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.015950+0000", 2024-04-24T14:33:41.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7116e-05 2024-04-24T14:33:41.272 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.015974+0000", 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3666999999999999e-05 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.017121+0000", 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011470040000000001 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.017214+0000", 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2651000000000005e-05 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.273 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.274 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.274 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4995.0:12 3.1 e53/18 3:990e5750:::rbd_header.12ff64619e2d:head v 53'18, mlcod=53'16)", 2024-04-24T14:33:41.274 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.228370+0000", 2024-04-24T14:33:41.274 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.033756299, 2024-04-24T14:33:41.274 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016640209999999999, 2024-04-24T14:33:41.274 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.274 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:41.274 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.228370+0000", 2024-04-24T14:33:41.274 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.228370+0000", 2024-04-24T14:33:41.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.228368+0000", 2024-04-24T14:33:41.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:41.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.228383+0000", 2024-04-24T14:33:41.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5656000000000001e-05 2024-04-24T14:33:41.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.276 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.276 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.228416+0000", 2024-04-24T14:33:41.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2638999999999997e-05 2024-04-24T14:33:41.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.276 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.276 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.228430+0000", 2024-04-24T14:33:41.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.418e-05 2024-04-24T14:33:41.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.228574+0000", 2024-04-24T14:33:41.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014421 2024-04-24T14:33:41.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:41.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.228636+0000", 2024-04-24T14:33:41.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2038999999999993e-05 2024-04-24T14:33:41.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:41.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.229923+0000", 2024-04-24T14:33:41.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012861529999999999 2024-04-24T14:33:41.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.278 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.278 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.278 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.230034+0000", 2024-04-24T14:33:41.278 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011158399999999999 2024-04-24T14:33:41.278 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.278 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.278 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.278 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.278 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.278 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4995.0:15 3.1 e53/18 3:990e5750:::rbd_header.12ff64619e2d:head v 53'19, mlcod=53'17)", 2024-04-24T14:33:41.278 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.260466+0000", 2024-04-24T14:33:41.278 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.0016608410000001, 2024-04-24T14:33:41.278 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015691640000000001, 2024-04-24T14:33:41.278 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.279 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:41.279 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.260466+0000", 2024-04-24T14:33:41.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.260466+0000", 2024-04-24T14:33:41.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.260464+0000", 2024-04-24T14:33:41.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:41.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.260478+0000", 2024-04-24T14:33:41.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3623e-05 2024-04-24T14:33:41.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.260509+0000", 2024-04-24T14:33:41.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0907000000000001e-05 2024-04-24T14:33:41.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.260524+0000", 2024-04-24T14:33:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5761999999999999e-05 2024-04-24T14:33:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.260654+0000", 2024-04-24T14:33:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000129541 2024-04-24T14:33:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.260720+0000", 2024-04-24T14:33:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5704999999999995e-05 2024-04-24T14:33:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:41.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.261917+0000", 2024-04-24T14:33:41.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011978920000000001 2024-04-24T14:33:41.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.262035+0000", 2024-04-24T14:33:41.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117404 2024-04-24T14:33:41.282 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.282 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.282 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.283 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "pg_update_log_missing(3.1 epoch 53/18 rep_tid 36 entries 53'20 (0'0) error 3:990e5750:::rbd_header.12ff64619e2d:head by client.4772.0:93 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'18)", 2024-04-24T14:33:41.283 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.266843+0000", 2024-04-24T14:33:41.283 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.99528358400000005, 2024-04-24T14:33:41.283 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013429379999999999, 2024-04-24T14:33:41.283 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.283 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:41.283 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.283 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.283 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.266843+0000", 2024-04-24T14:33:41.283 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.283 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.266843+0000", 2024-04-24T14:33:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.266841+0000", 2024-04-24T14:33:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.266861+0000", 2024-04-24T14:33:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9741999999999998e-05 2024-04-24T14:33:41.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.266902+0000", 2024-04-24T14:33:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1022999999999997e-05 2024-04-24T14:33:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.266917+0000", 2024-04-24T14:33:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5166e-05 2024-04-24T14:33:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.267072+0000", 2024-04-24T14:33:41.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015511 2024-04-24T14:33:41.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.286 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.286 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.268186+0000", 2024-04-24T14:33:41.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001113527 2024-04-24T14:33:41.286 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.286 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.286 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.286 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4772.0:156 3.1 e53/18 3:990e5750:::rbd_header.12ff64619e2d:head v 53'21, mlcod=53'19)", 2024-04-24T14:33:41.286 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.353098+0000", 2024-04-24T14:33:41.286 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.90902846999999998, 2024-04-24T14:33:41.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016991879999999999, 2024-04-24T14:33:41.287 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.287 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:41.287 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.353098+0000", 2024-04-24T14:33:41.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.353098+0000", 2024-04-24T14:33:41.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.353096+0000", 2024-04-24T14:33:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.353110+0000", 2024-04-24T14:33:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4192999999999999e-05 2024-04-24T14:33:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.353144+0000", 2024-04-24T14:33:41.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3423000000000002e-05 2024-04-24T14:33:41.289 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.289 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.289 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.353155+0000", 2024-04-24T14:33:41.289 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1355999999999999e-05 2024-04-24T14:33:41.289 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.289 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.289 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.353286+0000", 2024-04-24T14:33:41.289 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000130283 2024-04-24T14:33:41.289 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.289 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:41.289 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.353338+0000", 2024-04-24T14:33:41.290 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2497000000000002e-05 2024-04-24T14:33:41.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.290 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.290 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:41.290 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.354692+0000", 2024-04-24T14:33:41.290 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013535839999999999 2024-04-24T14:33:41.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.290 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.290 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.290 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.354797+0000", 2024-04-24T14:33:41.290 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105622 2024-04-24T14:33:41.290 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.290 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.290 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.291 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5007.0:12 3.1 e53/18 3:990e5750:::rbd_header.12ff64619e2d:head v 53'22, mlcod=53'20)", 2024-04-24T14:33:41.291 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.016983+0000", 2024-04-24T14:33:41.291 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.245143112, 2024-04-24T14:33:41.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016028080000000001, 2024-04-24T14:33:41.291 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.291 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:41.291 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.291 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.291 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.016983+0000", 2024-04-24T14:33:41.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.016983+0000", 2024-04-24T14:33:41.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.016982+0000", 2024-04-24T14:33:41.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:41.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.016997+0000", 2024-04-24T14:33:41.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4916000000000001e-05 2024-04-24T14:33:41.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.017022+0000", 2024-04-24T14:33:41.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5310999999999999e-05 2024-04-24T14:33:41.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.017031+0000", 2024-04-24T14:33:41.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6440000000000006e-06 2024-04-24T14:33:41.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.294 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.017161+0000", 2024-04-24T14:33:41.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013077799999999999 2024-04-24T14:33:41.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.294 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:41.294 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.017229+0000", 2024-04-24T14:33:41.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7285000000000004e-05 2024-04-24T14:33:41.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.294 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:41.294 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.018482+0000", 2024-04-24T14:33:41.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001253087 2024-04-24T14:33:41.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.018586+0000", 2024-04-24T14:33:41.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000104378 2024-04-24T14:33:41.295 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.295 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.295 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.295 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.295 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.295 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:41.296 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:41.320 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records in 2024-04-24T14:33:41.320 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records out 2024-04-24T14:33:41.320 INFO:tasks.workunit.client.0.smithi173.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.129762 s, 517 MB/s 2024-04-24T14:33:41.321 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo dd if=/dev/nbd0 bs=1M 2024-04-24T14:33:41.321 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:33:41.321 INFO:tasks.workunit.client.0.smithi173.stderr:++ md5sum 2024-04-24T14:33:41.322 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:33:41.323 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:33:41.324 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which dd 2024-04-24T14:33:41.325 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/dd 2024-04-24T14:33:41.325 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:33:41.325 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/dd if=/dev/nbd0 bs=1M 2024-04-24T14:33:41.329 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:41.331+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:41.398 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:41.398 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:41.398 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:41.399 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:41.407 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:41.430 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:41.430+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:41.507 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:41.507 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:41.507 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:41.507 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:41.507 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:41.516 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:41.531 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:41.535+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:41.612 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:41.612 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:41.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:41.612 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:41.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.612 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:1 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:41.613 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:41.613 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.415931379, 2024-04-24T14:33:41.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002159392, 2024-04-24T14:33:41.613 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.613 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:41.613 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:41.613 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:41.613 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:41.613 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 1 2024-04-24T14:33:41.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.613 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:41.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:41.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201710+0000", 2024-04-24T14:33:41.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:33:41.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201731+0000", 2024-04-24T14:33:41.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0835999999999998e-05 2024-04-24T14:33:41.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201769+0000", 2024-04-24T14:33:41.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7472999999999998e-05 2024-04-24T14:33:41.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201796+0000", 2024-04-24T14:33:41.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6798999999999999e-05 2024-04-24T14:33:41.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201923+0000", 2024-04-24T14:33:41.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012770699999999999 2024-04-24T14:33:41.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203873+0000", 2024-04-24T14:33:41.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00194992 2024-04-24T14:33:41.616 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.616 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.616 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.617 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:41.617 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:41.617 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.414365642, 2024-04-24T14:33:41.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026872300000000001, 2024-04-24T14:33:41.617 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.617 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:41.617 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:41.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:41.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203277+0000", 2024-04-24T14:33:41.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203292+0000", 2024-04-24T14:33:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4277000000000001e-05 2024-04-24T14:33:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203327+0000", 2024-04-24T14:33:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5219000000000003e-05 2024-04-24T14:33:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203350+0000", 2024-04-24T14:33:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2807e-05 2024-04-24T14:33:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203437+0000", 2024-04-24T14:33:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7070999999999994e-05 2024-04-24T14:33:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203548+0000", 2024-04-24T14:33:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000111576 2024-04-24T14:33:41.620 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:2 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.414252337000001, 2024-04-24T14:33:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047788300000000002, 2024-04-24T14:33:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203391+0000", 2024-04-24T14:33:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203400+0000", 2024-04-24T14:33:41.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5860000000000005e-06 2024-04-24T14:33:41.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203421+0000", 2024-04-24T14:33:41.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1056e-05 2024-04-24T14:33:41.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203434+0000", 2024-04-24T14:33:41.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3093e-05 2024-04-24T14:33:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203623+0000", 2024-04-24T14:33:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000188646 2024-04-24T14:33:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203871+0000", 2024-04-24T14:33:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00024812199999999998 2024-04-24T14:33:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.625 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:41.625 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:41.625 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.414194932000001, 2024-04-24T14:33:41.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059468100000000001, 2024-04-24T14:33:41.625 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.625 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:41.625 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:41.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:41.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203449+0000", 2024-04-24T14:33:41.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:41.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203469+0000", 2024-04-24T14:33:41.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0003000000000001e-05 2024-04-24T14:33:41.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203486+0000", 2024-04-24T14:33:41.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7309999999999999e-05 2024-04-24T14:33:41.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203501+0000", 2024-04-24T14:33:41.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.49e-05 2024-04-24T14:33:41.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203923+0000", 2024-04-24T14:33:41.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042195600000000002 2024-04-24T14:33:41.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204045+0000", 2024-04-24T14:33:41.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012202200000000001 2024-04-24T14:33:41.628 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.628 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.628 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.629 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:3 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:41.629 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:41.629 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.412824004000001, 2024-04-24T14:33:41.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00093837399999999996, 2024-04-24T14:33:41.629 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.629 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:41.629 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:41.629 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:41.629 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:41.629 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 3 2024-04-24T14:33:41.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.629 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:41.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:41.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204819+0000", 2024-04-24T14:33:41.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:41.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204833+0000", 2024-04-24T14:33:41.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3346999999999999e-05 2024-04-24T14:33:41.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204866+0000", 2024-04-24T14:33:41.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3302000000000001e-05 2024-04-24T14:33:41.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204903+0000", 2024-04-24T14:33:41.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7576000000000001e-05 2024-04-24T14:33:41.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205040+0000", 2024-04-24T14:33:41.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013616 2024-04-24T14:33:41.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:41.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205231+0000", 2024-04-24T14:33:41.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019118999999999999 2024-04-24T14:33:41.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205610+0000", 2024-04-24T14:33:41.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037947000000000002 2024-04-24T14:33:41.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205760+0000", 2024-04-24T14:33:41.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000149259 2024-04-24T14:33:41.633 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.633 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.633 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.634 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e47/46 cost 1048576)", 2024-04-24T14:33:41.634 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.684271+0000", 2024-04-24T14:33:41.634 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.9333740640000006, 2024-04-24T14:33:41.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00025325, 2024-04-24T14:33:41.634 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.634 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:41.634 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684271+0000", 2024-04-24T14:33:41.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684271+0000", 2024-04-24T14:33:41.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684270+0000", 2024-04-24T14:33:41.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:41.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684288+0000", 2024-04-24T14:33:41.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7844000000000001e-05 2024-04-24T14:33:41.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684307+0000", 2024-04-24T14:33:41.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8759000000000001e-05 2024-04-24T14:33:41.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684315+0000", 2024-04-24T14:33:41.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9200000000000004e-06 2024-04-24T14:33:41.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684368+0000", 2024-04-24T14:33:41.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3409000000000003e-05 2024-04-24T14:33:41.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684524+0000", 2024-04-24T14:33:41.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015631299999999999 2024-04-24T14:33:41.637 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.637 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.637 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.638 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:118 3.2 e47/46 3:774f7988:ns::rbd_directory:head v 47'2, mlcod=0'0)", 2024-04-24T14:33:41.638 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.686690+0000", 2024-04-24T14:33:41.638 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.9309550059999996, 2024-04-24T14:33:41.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001318173, 2024-04-24T14:33:41.638 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.638 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:41.638 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686690+0000", 2024-04-24T14:33:41.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686690+0000", 2024-04-24T14:33:41.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686689+0000", 2024-04-24T14:33:41.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:41.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686698+0000", 2024-04-24T14:33:41.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8529999999999998e-06 2024-04-24T14:33:41.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686714+0000", 2024-04-24T14:33:41.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5698000000000001e-05 2024-04-24T14:33:41.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686721+0000", 2024-04-24T14:33:41.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7719999999999997e-06 2024-04-24T14:33:41.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686786+0000", 2024-04-24T14:33:41.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5333000000000006e-05 2024-04-24T14:33:41.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:41.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686818+0000", 2024-04-24T14:33:41.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1998000000000001e-05 2024-04-24T14:33:41.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:41.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.687962+0000", 2024-04-24T14:33:41.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011436980000000001 2024-04-24T14:33:41.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.688008+0000", 2024-04-24T14:33:41.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6718999999999998e-05 2024-04-24T14:33:41.642 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.642 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.642 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.642 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:119 3.2 e47/46 3:684f7435:::rbd_id.testrbdnbd16809:head v 47'3, mlcod=37'1)", 2024-04-24T14:33:41.643 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.738094+0000", 2024-04-24T14:33:41.643 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.8795515229999999, 2024-04-24T14:33:41.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016595679999999999, 2024-04-24T14:33:41.643 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.643 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:41.643 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738094+0000", 2024-04-24T14:33:41.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738094+0000", 2024-04-24T14:33:41.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738092+0000", 2024-04-24T14:33:41.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:41.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738116+0000", 2024-04-24T14:33:41.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3883000000000002e-05 2024-04-24T14:33:41.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738147+0000", 2024-04-24T14:33:41.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1269000000000002e-05 2024-04-24T14:33:41.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738159+0000", 2024-04-24T14:33:41.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1443e-05 2024-04-24T14:33:41.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738290+0000", 2024-04-24T14:33:41.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013122399999999999 2024-04-24T14:33:41.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:41.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738361+0000", 2024-04-24T14:33:41.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1062000000000005e-05 2024-04-24T14:33:41.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:41.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.739670+0000", 2024-04-24T14:33:41.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013087540000000001 2024-04-24T14:33:41.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.739753+0000", 2024-04-24T14:33:41.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3607000000000001e-05 2024-04-24T14:33:41.647 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.647 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.647 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.647 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:120 3.2 e47/46 3:5cea7035:::rbd_info:head v 47'4, mlcod=47'2)", 2024-04-24T14:33:41.647 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.745883+0000", 2024-04-24T14:33:41.647 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.8717622110000001, 2024-04-24T14:33:41.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0014160220000000001, 2024-04-24T14:33:41.647 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.648 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:41.648 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745883+0000", 2024-04-24T14:33:41.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745883+0000", 2024-04-24T14:33:41.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745881+0000", 2024-04-24T14:33:41.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:41.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745897+0000", 2024-04-24T14:33:41.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5519000000000002e-05 2024-04-24T14:33:41.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745952+0000", 2024-04-24T14:33:41.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5071999999999997e-05 2024-04-24T14:33:41.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745968+0000", 2024-04-24T14:33:41.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6192999999999999e-05 2024-04-24T14:33:41.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.746085+0000", 2024-04-24T14:33:41.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117187 2024-04-24T14:33:41.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:41.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.746137+0000", 2024-04-24T14:33:41.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2513999999999999e-05 2024-04-24T14:33:41.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:41.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.747239+0000", 2024-04-24T14:33:41.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00110176 2024-04-24T14:33:41.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.747299+0000", 2024-04-24T14:33:41.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9703000000000001e-05 2024-04-24T14:33:41.652 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.652 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.652 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.652 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:33:41.652 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:41.652 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.545713396, 2024-04-24T14:33:41.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:33:41.652 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:41.653 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:41.653 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:41.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:41.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:41.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:41.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:41.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:41.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:41.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:33:41.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:41.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:41.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:33:41.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:33:41.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:41.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:33:41.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:33:41.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:33:41.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:33:41.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:41.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:33:41.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:33:41.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:41.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:33:41.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:33:41.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:41.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:33:41.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:33:41.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:41.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:41.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:41.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:33:41.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:33:41.657 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.657 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.657 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.657 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:41.657 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:41.657 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:41.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:41.659 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:41.663+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:41.741 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-24T14:33:41.758 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:41.763+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:41.779 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:41.779 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:41.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.779 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:41.779 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:41.779 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.028144135000000001, 2024-04-24T14:33:41.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028179321, 2024-04-24T14:33:41.779 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:41.779 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for rw locks", 2024-04-24T14:33:41.780 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:41.780 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:41.780 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:41.780 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:33:41.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.780 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:41.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:41.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:41.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:41.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:33:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:33:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:33:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:41.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:33:41.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:33:41.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:33:41.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:33:41.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:33:41.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:33:41.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:41.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:33:41.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:33:41.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:33:41.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:33:41.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:41.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:33:41.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:33:41.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.784 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:41.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.784 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:41.784 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:41.784 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.028093301000000001, 2024-04-24T14:33:41.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028240324000000001, 2024-04-24T14:33:41.784 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:41.785 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for rw locks", 2024-04-24T14:33:41.785 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:41.785 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:41.785 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:41.785 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:33:41.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.785 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:41.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:41.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:41.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:41.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:41.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:41.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:41.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:41.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:41.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:33:41.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:33:41.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:41.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:33:41.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:33:41.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:33:41.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:33:41.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:33:41.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:33:41.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:41.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:33:41.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:33:41.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:33:41.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:33:41.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:41.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:33:41.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:33:41.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.789 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:41.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.789 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:41.789 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:41.790 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.028056993999999998, 2024-04-24T14:33:41.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028320834, 2024-04-24T14:33:41.790 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:41.790 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for rw locks", 2024-04-24T14:33:41.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:41.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:41.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:41.790 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:33:41.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.790 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:41.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:41.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:41.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:41.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:41.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:41.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:33:41.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:41.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:41.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:33:41.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:33:41.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:41.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:33:41.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:33:41.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:33:41.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:33:41.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:33:41.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:33:41.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:41.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:33:41.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:33:41.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:33:41.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:33:41.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:41.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:33:41.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:33:41.794 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.794 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:41.794 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.794 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:41.794 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:41.794 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.024645744000000001, 2024-04-24T14:33:41.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025006502999999999, 2024-04-24T14:33:41.795 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:41.795 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for rw locks", 2024-04-24T14:33:41.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:41.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:41.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:41.795 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:33:41.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.795 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:41.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:41.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:41.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:41.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:41.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:41.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:33:41.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:41.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:41.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:33:41.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:33:41.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:41.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:33:41.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:33:41.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:33:41.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:33:41.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:33:41.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:33:41.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:41.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:33:41.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:33:41.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:33:41.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:33:41.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:41.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:33:41.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:33:41.799 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.799 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:41.799 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.800 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:41.800 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:41.800 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.024584067000000001, 2024-04-24T14:33:41.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025049259000000001, 2024-04-24T14:33:41.800 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:41.800 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for rw locks", 2024-04-24T14:33:41.800 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:41.800 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:41.800 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:41.800 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:33:41.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.800 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:41.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:41.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:41.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:41.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:41.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:41.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:33:41.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:41.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:41.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:33:41.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:33:41.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:41.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:33:41.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:33:41.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:33:41.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:33:41.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:33:41.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:33:41.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:41.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:33:41.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:33:41.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:33:41.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:33:41.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:41.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:33:41.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:33:41.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.805 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:41.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.805 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:41.805 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:41.805 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.024549491, 2024-04-24T14:33:41.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025110037000000002, 2024-04-24T14:33:41.805 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:41.805 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for rw locks", 2024-04-24T14:33:41.805 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:41.805 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:41.805 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:41.806 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:33:41.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.806 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:41.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:41.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:41.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:41.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:41.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:41.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:41.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:41.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:41.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:33:41.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:33:41.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:41.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:33:41.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:33:41.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:33:41.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:33:41.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:33:41.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:33:41.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:41.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:33:41.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:33:41.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:33:41.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:33:41.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:41.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:33:41.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:33:41.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.810 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:41.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.810 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:41.810 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:41.810 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.024520152, 2024-04-24T14:33:41.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025178059999999999, 2024-04-24T14:33:41.810 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:41.810 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for rw locks", 2024-04-24T14:33:41.810 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:33:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:41.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:33:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:33:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:33:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:33:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:33:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:33:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:33:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:33:41.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:33:41.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:41.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:33:41.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:33:41.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:33:41.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:33:41.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:41.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:33:41.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:33:41.815 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.815 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:41.815 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.815 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:41.815 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:41.815 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.024443570000000001, 2024-04-24T14:33:41.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025196422999999999, 2024-04-24T14:33:41.816 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:41.816 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for rw locks", 2024-04-24T14:33:41.816 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:41.816 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:41.816 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:41.816 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:33:41.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.816 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:41.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:41.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:41.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:41.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:41.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:41.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:33:41.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:41.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:41.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:33:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:33:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:33:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:33:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:33:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:33:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:33:41.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:33:41.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:41.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:33:41.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:33:41.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:33:41.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:33:41.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:41.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:33:41.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:33:41.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.820 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:41.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.820 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:41.820 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:41.821 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.024396024999999998, 2024-04-24T14:33:41.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025272328, 2024-04-24T14:33:41.821 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:41.821 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for rw locks", 2024-04-24T14:33:41.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:41.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:41.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:41.821 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:33:41.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.821 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:41.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:41.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:41.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:41.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:41.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:41.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:33:41.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:41.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:33:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:33:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:33:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:33:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:33:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:33:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:33:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:33:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:33:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:33:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:33:41.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:33:41.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:41.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:33:41.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:33:41.825 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.825 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:41.825 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.826 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:41.826 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:41.826 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.024355383000000001, 2024-04-24T14:33:41.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025326962000000001, 2024-04-24T14:33:41.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:41.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for rw locks", 2024-04-24T14:33:41.826 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:41.826 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:41.826 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:41.826 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:33:41.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.826 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:41.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:41.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:41.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:41.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:41.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:41.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:41.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:41.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:41.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:33:41.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:33:41.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:41.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:33:41.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:33:41.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:33:41.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:33:41.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:33:41.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:33:41.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:41.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:33:41.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:33:41.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:33:41.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:33:41.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:41.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:33:41.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:33:41.830 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.830 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:41.830 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.831 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:41.831 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:41.831 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.024338272000000001, 2024-04-24T14:33:41.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025403578, 2024-04-24T14:33:41.831 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:41.831 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for rw locks", 2024-04-24T14:33:41.831 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:41.831 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:41.831 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:41.831 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:33:41.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.832 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:41.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:41.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:41.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:41.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:41.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:41.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:41.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:33:41.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:41.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:41.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:33:41.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:33:41.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:41.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:33:41.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:33:41.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:41.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:33:41.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:33:41.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:33:41.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:33:41.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:41.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:33:41.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:33:41.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:41.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:33:41.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:33:41.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:41.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:41.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:41.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:33:41.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:33:41.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.835 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:41.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.836 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:41.836 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:41.836 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 11 2024-04-24T14:33:41.836 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:41.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:41.859 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:41.862+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:41.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:33:41.949 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records in 2024-04-24T14:33:41.950 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records out 2024-04-24T14:33:41.950 INFO:tasks.workunit.client.0.smithi173.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.615495 s, 109 MB/s 2024-04-24T14:33:41.950 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:41.950 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:41.951 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:41.951 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:41.951 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:41.951 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' '2db8a66dcaa876d1d91e96c113fc1f74 -' = '2db8a66dcaa876d1d91e96c113fc1f74 -' ']' 2024-04-24T14:33:41.951 INFO:tasks.workunit.client.0.smithi173.stderr:+ dd if=/dev/urandom of=/tmp/tmp.waDYLkcsIV/data bs=1M count=64 2024-04-24T14:33:41.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:41.960 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:41.962+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:42.017 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:42.035 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:42.054 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:42.054 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:42.057 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:42.061 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:42.063+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:42.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:42.078 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:42.098 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:42.099 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:42.099 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:42.099 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:42.099 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:42.099 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:42.099 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:42.102 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:42.102 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:42.157 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:42.162 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:42.166+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:42.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:42.262 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:42.262 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:42.263 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:42.266+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:42.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:42.363 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:42.367+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:42.371 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:42.371 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:42.380 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:42.464 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:42.467+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:42.487 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:42.487 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:42.497 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:42.547 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records in 2024-04-24T14:33:42.547 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records out 2024-04-24T14:33:42.547 INFO:tasks.workunit.client.0.smithi173.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.580961 s, 116 MB/s 2024-04-24T14:33:42.548 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M oflag=direct 2024-04-24T14:33:42.548 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:42.548 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:42.550 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:42.551 INFO:tasks.workunit.client.0.smithi173.stderr:++ which dd 2024-04-24T14:33:42.553 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/dd 2024-04-24T14:33:42.553 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:42.553 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M oflag=direct 2024-04-24T14:33:42.565 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:42.567+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:42.607 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:42.608 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:42.618 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:42.666 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:42.666+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:42.718 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:42.718 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:42.722 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [1] live_osds: [0, 2, 3, 4, 5] 2024-04-24T14:33:42.722 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:33:42.722 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-24T14:33:42.722 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.889637 2024-04-24T14:33:42.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 0.8896365438549089 2024-04-24T14:33:42.727 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:42.770 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:42.775+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:42.832 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:42.832 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:42.843 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:42.867 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:42.871+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:42.946 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:42.947 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:42.960 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:42.968 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:42.971+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:43.062 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:43.063 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:43.069 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:43.071+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:43.073 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:43.170 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:43.171+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:43.175 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:43.175 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:43.185 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:43.270 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:43.270+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:43.280 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records in 2024-04-24T14:33:43.280 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records out 2024-04-24T14:33:43.280 INFO:tasks.workunit.client.0.smithi173.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.716364 s, 93.7 MB/s 2024-04-24T14:33:43.283 INFO:tasks.workunit.client.0.smithi173.stderr:++ dd if=/tmp/tmp.waDYLkcsIV/data bs=1M 2024-04-24T14:33:43.283 INFO:tasks.workunit.client.0.smithi173.stderr:++ md5sum 2024-04-24T14:33:43.285 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:43.285 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:43.294 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:43.372 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:43.375+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:43.390 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:43.391 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:43.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:43.423 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records in 2024-04-24T14:33:43.423 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records out 2024-04-24T14:33:43.423 INFO:tasks.workunit.client.0.smithi173.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.138829 s, 483 MB/s 2024-04-24T14:33:43.424 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd -p rbd --no-progress export testrbdnbd16809 - 2024-04-24T14:33:43.425 INFO:tasks.workunit.client.0.smithi173.stderr:++ md5sum 2024-04-24T14:33:43.472 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:43.475+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:43.509 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:33:43.510+0000 7f913fec2700 0 -- 172.21.15.173:0/824234947 >> [v2:172.21.15.90:6808/20310,v1:172.21.15.90:6809/20310] conn(0x7f9118005270 msgr2=0x7f91180012c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T14:33:43.511 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:43.511 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:43.515 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:33:43.518+0000 7f91406c3700 0 -- 172.21.15.173:0/824234947 >> [v2:172.21.15.90:6816/20309,v1:172.21.15.90:6817/20309] conn(0x7f9124008980 msgr2=0x7f912400ae00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T14:33:43.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:43.573 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:43.574+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:43.613 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:43.613 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:43.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:43.664 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 'a5da6bff04bdca0e69b6231a1d498387 -' = 'a5da6bff04bdca0e69b6231a1d498387 -' ']' 2024-04-24T14:33:43.665 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device /dev/nbd0 17204 2024-04-24T14:33:43.665 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=/dev/nbd0 2024-04-24T14:33:43.665 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=17204 2024-04-24T14:33:43.665 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:33:43.665 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:43.665 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:43.666 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:43.667 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:33:43.669 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:33:43.669 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:43.669 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:33:43.674 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:43.674+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:43.717 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:43.717 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:43.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:43.745 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.2 primary-affinity to 0.889637 (8583032) 2024-04-24T14:33:43.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T14:33:43.759 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:33:43.759 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^17204\b' 2024-04-24T14:33:43.759 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^17204\b' 2024-04-24T14:33:43.775 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:43.779+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:43.816 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:43.817 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:33:43.817 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *17204\b' 2024-04-24T14:33:43.817 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *17204\b' 2024-04-24T14:33:43.823 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:43.824 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:33:43.826 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:43.826 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:43.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:43.876 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:43.879+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:43.927 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:43.927 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:43.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:43.977 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:43.978+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:44.078 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:44.078+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:44.082 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:44.082 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:44.091 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:44.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:33:44.178 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:44.183+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:44.201 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:44.201 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:44.210 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:44.236 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:44.255 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:44.276 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:44.279 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:44.283+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:44.297 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:44.310 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:44.310 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:44.310 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:44.310 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:44.310 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:44.317 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:44.318 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:44.318 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:44.318 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:44.318 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:44.318 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:44.318 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:44.320 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:44.321 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:44.321 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:44.326 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd --options notrim map rbd/testrbdnbd16809 2024-04-24T14:33:44.326 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:33:44.327 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:33:44.328 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:33:44.329 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:33:44.332 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:33:44.332 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:33:44.332 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options notrim map rbd/testrbdnbd16809 2024-04-24T14:33:44.380 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:44.383+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:44.425 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:44.425 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:44.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:44.425 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:44.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.426 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.426 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:44.426 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.101254978, 2024-04-24T14:33:44.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:33:44.426 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.426 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.426 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:33:44.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.426 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:44.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:44.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:33:44.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:44.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:33:44.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:33:44.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:33:44.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:33:44.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:33:44.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:33:44.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:33:44.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:33:44.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:33:44.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:33:44.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:33:44.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:33:44.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:33:44.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:33:44.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:33:44.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:33:44.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:33:44.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:33:44.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:33:44.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:33:44.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.431 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.432 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.432 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:44.432 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.801740035999998, 2024-04-24T14:33:44.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:33:44.432 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.432 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.432 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:33:44.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.432 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:33:44.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:33:44.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:33:44.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:33:44.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:33:44.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:33:44.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:33:44.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:33:44.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:33:44.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:33:44.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:33:44.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:33:44.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:33:44.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:33:44.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:33:44.437 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.437 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.437 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.437 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.437 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:44.437 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.801514268000005, 2024-04-24T14:33:44.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:33:44.438 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.438 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.438 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:33:44.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.438 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:44.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:44.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:33:44.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:44.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:33:44.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:33:44.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:33:44.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:33:44.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:33:44.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:33:44.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:33:44.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:33:44.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:33:44.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:33:44.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:33:44.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:33:44.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:33:44.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:33:44.443 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.443 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.443 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.443 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.443 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:44.443 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.801337180999994, 2024-04-24T14:33:44.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:44.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:33:44.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:44.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:33:44.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:33:44.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:33:44.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:33:44.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:33:44.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:33:44.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:33:44.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:33:44.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:33:44.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:33:44.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:33:44.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:33:44.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:33:44.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.801233267000001, 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.449 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.450 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:33:44.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.450 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:44.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:44.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:33:44.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:44.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:33:44.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:33:44.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:33:44.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:33:44.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:33:44.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:33:44.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:33:44.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:33:44.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:33:44.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:33:44.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:33:44.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:33:44.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:33:44.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:33:44.454 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.454 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.455 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.455 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.455 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:44.455 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.800936946999997, 2024-04-24T14:33:44.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:33:44.455 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.455 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.455 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.455 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.455 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.455 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:33:44.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:33:44.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:44.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:33:44.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:33:44.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:33:44.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:33:44.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:33:44.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:33:44.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:33:44.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:33:44.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:33:44.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:33:44.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:33:44.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:33:44.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:33:44.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:33:44.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:33:44.460 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.460 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.460 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.461 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.461 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:44.461 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.800844815999994, 2024-04-24T14:33:44.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:33:44.461 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.461 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.461 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.461 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.461 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.461 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:33:44.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.461 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:44.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:44.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:33:44.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:44.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:33:44.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:33:44.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:33:44.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:33:44.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:33:44.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:33:44.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:33:44.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:33:44.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:33:44.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:33:44.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:33:44.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:33:44.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:33:44.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:33:44.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:33:44.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:33:44.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:33:44.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:33:44.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:33:44.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:33:44.466 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.466 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.466 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.466 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.466 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:44.466 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.800759604000007, 2024-04-24T14:33:44.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:33:44.467 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.467 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.467 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.467 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.467 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.467 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:33:44.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.467 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:44.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:44.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:33:44.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:44.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:33:44.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:33:44.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:33:44.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:33:44.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:33:44.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:33:44.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:33:44.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:33:44.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:33:44.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:33:44.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:33:44.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:33:44.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:33:44.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.800522723, 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.472 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.473 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.473 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.473 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:33:44.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.473 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:44.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:44.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:33:44.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:44.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:33:44.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:33:44.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:33:44.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:33:44.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:33:44.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:33:44.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:33:44.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:33:44.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.477 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.478 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.478 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:44.478 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.800441200999998, 2024-04-24T14:33:44.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:33:44.478 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.478 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.478 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:33:44.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.478 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:44.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:33:44.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:33:44.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:33:44.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:33:44.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:33:44.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:33:44.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:33:44.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:33:44.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:33:44.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:33:44.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:33:44.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:33:44.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:33:44.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:33:44.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:33:44.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:33:44.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:33:44.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:33:44.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:33:44.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:33:44.483 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.483 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.483 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.483 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.483 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:44.484 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.368682789999994, 2024-04-24T14:33:44.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:33:44.484 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.484 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.484 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.484 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.484 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.484 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:33:44.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.484 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:44.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:44.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:33:44.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:44.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:33:44.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:33:44.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:33:44.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:33:44.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:33:44.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:33:44.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:33:44.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:33:44.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:33:44.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:33:44.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:33:44.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.368641287000003, 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:33:44.489 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.490 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.490 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.490 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.490 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.490 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:33:44.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.490 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:44.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:33:44.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:33:44.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:33:44.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:33:44.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:33:44.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:33:44.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:33:44.494 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.495 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.495 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.495 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.495 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:44.495 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.368528854000004, 2024-04-24T14:33:44.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:33:44.495 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.495 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.495 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.495 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.495 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.495 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:33:44.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.496 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:44.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:44.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:33:44.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:33:44.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:33:44.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:33:44.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:33:44.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:33:44.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:33:44.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:33:44.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:33:44.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:33:44.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:33:44.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:33:44.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:33:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:33:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:33:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:33:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.368500553999993, 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:44.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:44.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:33:44.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:44.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:33:44.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:33:44.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:33:44.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:33:44.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:33:44.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:33:44.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:33:44.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:33:44.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:33:44.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:33:44.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:33:44.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:33:44.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:33:44.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:33:44.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:33:44.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:33:44.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:33:44.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:33:44.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:33:44.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:33:44.506 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.506 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.506 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.506 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.506 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:44.506 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.368412849999999, 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:44.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:33:44.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:44.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:33:44.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:33:44.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:33:44.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:33:44.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:33:44.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:33:44.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:33:44.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:33:44.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:33:44.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:33:44.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:33:44.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:33:44.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:33:44.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:33:44.512 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.512 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.512 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.512 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:44.512 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:44.512 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.368290048999995, 2024-04-24T14:33:44.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:33:44.512 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.512 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.512 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.513 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.513 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.513 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:33:44.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.513 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:44.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:44.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:33:44.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:44.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:33:44.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:33:44.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:33:44.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:33:44.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:33:44.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:33:44.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:33:44.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:33:44.517 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.518 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.518 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.518 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:33:44.518 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:44.518 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 80.725050116999995, 2024-04-24T14:33:44.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:33:44.518 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.518 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:44.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:33:44.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:44.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:33:44.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:33:44.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:33:44.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:33:44.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:33:44.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:33:44.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:33:44.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:33:44.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:33:44.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:33:44.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:44.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:33:44.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:33:44.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:44.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:33:44.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:33:44.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:44.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:33:44.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:33:44.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:33:44.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:33:44.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:33:44.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:33:44.523 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.523 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.523 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:33:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.062045489000001, 2024-04-24T14:33:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:33:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:33:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:33:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:33:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.524 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:44.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:44.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:33:44.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:44.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:33:44.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:33:44.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:33:44.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:33:44.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:33:44.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:33:44.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:33:44.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:33:44.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:33:44.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:33:44.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:33:44.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.529 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.529 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:44.529 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.3679214899999996, 2024-04-24T14:33:44.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:33:44.529 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.529 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.529 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.529 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:33:44.529 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:33:44.529 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:33:44.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.529 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:44.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:44.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:33:44.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:33:44.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:33:44.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:33:44.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:44.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:33:44.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:33:44.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:33:44.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:33:44.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:44.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:33:44.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:33:44.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:33:44.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:33:44.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:33:44.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:33:44.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:33:44.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:33:44.534 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.534 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.534 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.534 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:577 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:44.535 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.923702+0000", 2024-04-24T14:33:44.535 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 2.503615747, 2024-04-24T14:33:44.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012522472, 2024-04-24T14:33:44.535 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.535 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.535 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.535 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.535 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.535 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 577 2024-04-24T14:33:44.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.535 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923702+0000", 2024-04-24T14:33:44.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923702+0000", 2024-04-24T14:33:44.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923701+0000", 2024-04-24T14:33:44.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:44.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923708+0000", 2024-04-24T14:33:44.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.233e-06 2024-04-24T14:33:44.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923723+0000", 2024-04-24T14:33:44.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5099000000000001e-05 2024-04-24T14:33:44.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923733+0000", 2024-04-24T14:33:44.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:33:44.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.935366+0000", 2024-04-24T14:33:44.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011632105 2024-04-24T14:33:44.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.935446+0000", 2024-04-24T14:33:44.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012000000000006e-05 2024-04-24T14:33:44.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.936224+0000", 2024-04-24T14:33:44.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077877899999999995 2024-04-24T14:33:44.538 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.539 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.539 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.539 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.539 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.539 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:44.541 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:33:44.541 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:33:44.541 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:33:44.541 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:33:44.541 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:33:44.541 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:33:44.541 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:44.542 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:44.543+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:44.552 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=17315 2024-04-24T14:33:44.552 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 17315 2024-04-24T14:33:44.552 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 17315 -C rbd-nbd 2024-04-24T14:33:44.558 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:33:44.558 INFO:tasks.workunit.client.0.smithi173.stdout: 17315 ? 00:00:00 rbd-nbd 2024-04-24T14:33:44.559 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd -p rbd --format xml du testrbdnbd16809 2024-04-24T14:33:44.560 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-04-24T14:33:44.635 INFO:tasks.workunit.client.0.smithi173.stderr:+ provisioned=67108864 2024-04-24T14:33:44.636 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd -p rbd --format xml du testrbdnbd16809 2024-04-24T14:33:44.636 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-04-24T14:33:44.641 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:44.642+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:44.652 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:44.652 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:44.653 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:44.653 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:44.662 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:44.710 INFO:tasks.workunit.client.0.smithi173.stderr:+ used=67108864 2024-04-24T14:33:44.710 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-04-24T14:33:44.710 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false _sudo blkdiscard /dev/nbd0 2024-04-24T14:33:44.710 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo blkdiscard /dev/nbd0 2024-04-24T14:33:44.710 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:44.711 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:44.712 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:44.713 INFO:tasks.workunit.client.0.smithi173.stderr:++ which blkdiscard 2024-04-24T14:33:44.715 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T14:33:44.715 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:44.715 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-04-24T14:33:44.740 INFO:tasks.workunit.client.0.smithi173.stderr:blkdiscard: /dev/nbd0: BLKDISCARD ioctl failed: Operation not supported 2024-04-24T14:33:44.741 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:44.742+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:44.742 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:44.742 INFO:tasks.workunit.client.0.smithi173.stderr:+ sync 2024-04-24T14:33:44.749 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-24T14:33:44.765 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:44.765 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:44.765 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:44.765 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:44.765 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:44.774 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:44.842 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:44.842+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:44.922 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:44.922 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:44.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:44.922 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:44.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.923 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:379 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.923 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:44.923 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2627329619999998, 2024-04-24T14:33:44.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068078410000000002, 2024-04-24T14:33:44.923 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.923 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.923 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.923 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.923 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.923 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 379 2024-04-24T14:33:44.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.923 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:44.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:44.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:44.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:44.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664240+0000", 2024-04-24T14:33:44.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6539999999999999e-06 2024-04-24T14:33:44.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664245+0000", 2024-04-24T14:33:44.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2179999999999998e-06 2024-04-24T14:33:44.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664249+0000", 2024-04-24T14:33:44.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2279999999999998e-06 2024-04-24T14:33:44.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670245+0000", 2024-04-24T14:33:44.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059962430000000001 2024-04-24T14:33:44.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670400+0000", 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015535399999999999 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671046+0000", 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064591599999999996 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:380 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2626863350000002, 2024-04-24T14:33:44.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076160830000000001, 2024-04-24T14:33:44.928 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.928 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.928 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.928 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.928 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.928 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 380 2024-04-24T14:33:44.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.928 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:33:44.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:33:44.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664284+0000", 2024-04-24T14:33:44.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:44.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664287+0000", 2024-04-24T14:33:44.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9739999999999998e-06 2024-04-24T14:33:44.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664302+0000", 2024-04-24T14:33:44.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5214999999999999e-05 2024-04-24T14:33:44.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664307+0000", 2024-04-24T14:33:44.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6859999999999997e-06 2024-04-24T14:33:44.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671098+0000", 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067912479999999997 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671253+0000", 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015427400000000001 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671901+0000", 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064845600000000005 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.932 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.932 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:44.932 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2626567899999999, 2024-04-24T14:33:44.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:33:44.932 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.932 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.932 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.932 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.932 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.932 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:33:44.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.933 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:44.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:44.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:44.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:44.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:33:44.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:33:44.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:33:44.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:33:44.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:33:44.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:33:44.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:33:44.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:33:44.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:33:44.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:33:44.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:33:44.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:33:44.936 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.936 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.936 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.936 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:382 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.936 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:33:44.936 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2596553070000001, 2024-04-24T14:33:44.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067029910000000002, 2024-04-24T14:33:44.936 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.937 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.937 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.937 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.937 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.937 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 382 2024-04-24T14:33:44.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.937 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:33:44.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:33:44.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667315+0000", 2024-04-24T14:33:44.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:44.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667321+0000", 2024-04-24T14:33:44.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2099999999999998e-06 2024-04-24T14:33:44.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667336+0000", 2024-04-24T14:33:44.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4612e-05 2024-04-24T14:33:44.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667343+0000", 2024-04-24T14:33:44.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0119999999999999e-06 2024-04-24T14:33:44.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672957+0000", 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056139700000000002 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.673127+0000", 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017044599999999999 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674019+0000", 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00089165499999999999 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.940 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.941 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.941 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:44.941 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2596222639999999, 2024-04-24T14:33:44.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:33:44.941 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.941 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.941 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.941 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.941 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.941 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:33:44.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.941 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:44.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:44.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:44.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:44.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:33:44.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:33:44.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:33:44.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:33:44.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:33:44.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:33:44.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:33:44.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:33:44.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:33:44.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:33:44.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:33:44.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:33:44.945 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.945 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.945 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.945 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.945 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:44.945 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2595980779999998, 2024-04-24T14:33:44.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:33:44.946 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.946 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.946 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.946 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.946 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.946 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:33:44.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.946 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:44.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:33:44.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.949 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:33:44.950 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2595806939999998, 2024-04-24T14:33:44.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:33:44.950 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.950 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.950 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.950 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.950 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.950 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:33:44.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.950 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:33:44.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:33:44.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:33:44.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:33:44.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:33:44.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:33:44.953 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.953 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.954 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.954 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:399 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.954 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:33:44.954 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.245170559, 2024-04-24T14:33:44.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071317810000000002, 2024-04-24T14:33:44.954 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.954 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.954 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.954 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.954 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.954 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 399 2024-04-24T14:33:44.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.955 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:33:44.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:33:44.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681800+0000", 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681804+0000", 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.715e-06 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681816+0000", 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1851000000000001e-05 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681820+0000", 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6450000000000004e-06 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688059+0000", 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062390090000000002 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688196+0000", 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013721999999999999 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688933+0000", 2024-04-24T14:33:44.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073605000000000003 2024-04-24T14:33:44.958 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.958 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.958 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.958 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.958 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:33:44.958 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2451446019999999, 2024-04-24T14:33:44.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:33:44.958 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:33:44.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:33:44.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:44.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:33:44.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:33:44.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:33:44.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:33:44.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:33:44.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:33:44.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:33:44.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:33:44.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:33:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:33:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:33:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:33:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.962 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.963 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:33:44.963 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.245112303, 2024-04-24T14:33:44.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:33:44.963 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.963 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.963 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.963 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.963 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.963 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:33:44.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.963 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:33:44.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:33:44.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:33:44.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:33:44.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:33:44.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:33:44.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:33:44.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:33:44.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:33:44.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:33:44.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:33:44.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:33:44.966 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.966 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.967 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.967 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:414 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.967 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:33:44.967 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2309168929999998, 2024-04-24T14:33:44.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069877910000000001, 2024-04-24T14:33:44.967 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.967 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.967 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.967 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.967 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.967 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 414 2024-04-24T14:33:44.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.968 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:33:44.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:33:44.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696053+0000", 2024-04-24T14:33:44.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:44.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696058+0000", 2024-04-24T14:33:44.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7509999999999999e-06 2024-04-24T14:33:44.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696074+0000", 2024-04-24T14:33:44.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5743000000000002e-05 2024-04-24T14:33:44.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696093+0000", 2024-04-24T14:33:44.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9508e-05 2024-04-24T14:33:44.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702314+0000", 2024-04-24T14:33:44.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062207010000000004 2024-04-24T14:33:44.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702431+0000", 2024-04-24T14:33:44.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011723299999999999 2024-04-24T14:33:44.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703042+0000", 2024-04-24T14:33:44.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000610752 2024-04-24T14:33:44.971 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.971 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.971 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.971 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.971 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:33:44.971 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2308600260000002, 2024-04-24T14:33:44.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:33:44.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:33:44.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:44.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:33:44.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:33:44.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:33:44.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:33:44.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:33:44.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:33:44.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:33:44.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:33:44.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:33:44.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:33:44.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:33:44.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:33:44.975 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.975 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.975 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2308309259999999, 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:33:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:33:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:33:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:33:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:33:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:33:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:33:44.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:33:44.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:33:44.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:33:44.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:33:44.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2308049959999998, 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:33:44.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.981 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:44.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:44.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.982 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.982 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:44.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:44.982 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.982 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.982 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:33:44.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:33:44.982 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.982 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.982 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:33:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:33:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:33:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:33:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:33:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:33:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:429 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3538944~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:33:44.984 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2165235559999998, 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071866789999999996, 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 429 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:33:44.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710447+0000", 2024-04-24T14:33:44.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:44.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710451+0000", 2024-04-24T14:33:44.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5930000000000002e-06 2024-04-24T14:33:44.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710465+0000", 2024-04-24T14:33:44.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.411e-05 2024-04-24T14:33:44.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710470+0000", 2024-04-24T14:33:44.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3129999999999998e-06 2024-04-24T14:33:44.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716696+0000", 2024-04-24T14:33:44.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006225983 2024-04-24T14:33:44.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716901+0000", 2024-04-24T14:33:44.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020550200000000001 2024-04-24T14:33:44.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717634+0000", 2024-04-24T14:33:44.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073292099999999998 2024-04-24T14:33:44.988 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.988 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.988 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:430 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.215159248, 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068326970000000004, 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 430 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:33:44.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:33:44.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711810+0000", 2024-04-24T14:33:44.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:44.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711822+0000", 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2166e-05 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711856+0000", 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3816000000000003e-05 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711873+0000", 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6742999999999999e-05 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717693+0000", 2024-04-24T14:33:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058204579999999997 2024-04-24T14:33:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717864+0000", 2024-04-24T14:33:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017128299999999999 2024-04-24T14:33:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718645+0000", 2024-04-24T14:33:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078033699999999998 2024-04-24T14:33:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.992 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2150845320000001, 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:33:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:33:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:33:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:44.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:33:44.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:33:44.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:44.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:33:44.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:33:44.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:44.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:33:44.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:33:44.996 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.996 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:44.996 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2150525999999999, 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:44.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:44.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:45.000 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.2149850839999998, 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:45.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:33:45.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:45.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:33:45.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:45.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:45.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:33:45.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:33:45.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:45.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:33:45.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:33:45.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:33:45.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:33:45.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:45.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:33:45.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:33:45.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:45.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:33:45.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:33:45.004 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.004 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.004 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.005 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:33:45.005 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:45.005 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.1575722009999998, 2024-04-24T14:33:45.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:33:45.005 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:45.005 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:45.005 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:45.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:45.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:45.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:45.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:33:45.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:33:45.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:33:45.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:33:45.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:45.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:33:45.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:33:45.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.008 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.009 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.009 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.009 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:45.010 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:45.011 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:45.015+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:45.111 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:45.115+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:45.116 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:45.116 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:45.116 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:45.116 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:45.126 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:45.212 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:45.215+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:45.226 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:45.226 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:45.226 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:45.226 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:45.226 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:45.236 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:45.312 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:45.314+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:45.332 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:45.332 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:45.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:45.332 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:1 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.136349871, 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002159392, 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 1 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201710+0000", 2024-04-24T14:33:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:33:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201731+0000", 2024-04-24T14:33:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0835999999999998e-05 2024-04-24T14:33:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201769+0000", 2024-04-24T14:33:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7472999999999998e-05 2024-04-24T14:33:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201796+0000", 2024-04-24T14:33:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6798999999999999e-05 2024-04-24T14:33:45.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201923+0000", 2024-04-24T14:33:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012770699999999999 2024-04-24T14:33:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203873+0000", 2024-04-24T14:33:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00194992 2024-04-24T14:33:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.134784134, 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026872300000000001, 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:45.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203277+0000", 2024-04-24T14:33:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203292+0000", 2024-04-24T14:33:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4277000000000001e-05 2024-04-24T14:33:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:45.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203327+0000", 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5219000000000003e-05 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203350+0000", 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2807e-05 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203437+0000", 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7070999999999994e-05 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203548+0000", 2024-04-24T14:33:45.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000111576 2024-04-24T14:33:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:2 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.134670829000001, 2024-04-24T14:33:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047788300000000002, 2024-04-24T14:33:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203391+0000", 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:45.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203400+0000", 2024-04-24T14:33:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5860000000000005e-06 2024-04-24T14:33:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203421+0000", 2024-04-24T14:33:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1056e-05 2024-04-24T14:33:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203434+0000", 2024-04-24T14:33:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3093e-05 2024-04-24T14:33:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203623+0000", 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000188646 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203871+0000", 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00024812199999999998 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.134613424000001, 2024-04-24T14:33:45.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059468100000000001, 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:45.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203449+0000", 2024-04-24T14:33:45.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:45.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:45.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203469+0000", 2024-04-24T14:33:45.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0003000000000001e-05 2024-04-24T14:33:45.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:45.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203486+0000", 2024-04-24T14:33:45.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7309999999999999e-05 2024-04-24T14:33:45.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203501+0000", 2024-04-24T14:33:45.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.49e-05 2024-04-24T14:33:45.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:45.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203923+0000", 2024-04-24T14:33:45.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042195600000000002 2024-04-24T14:33:45.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:45.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204045+0000", 2024-04-24T14:33:45.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012202200000000001 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:3 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.133242496000001, 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00093837399999999996, 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 3 2024-04-24T14:33:45.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.348 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:45.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:45.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:45.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:45.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:45.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:45.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204819+0000", 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204833+0000", 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3346999999999999e-05 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204866+0000", 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3302000000000001e-05 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204903+0000", 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7576000000000001e-05 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205040+0000", 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013616 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205231+0000", 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019118999999999999 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:45.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205610+0000", 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037947000000000002 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205760+0000", 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000149259 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e47/46 cost 1048576)", 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.684271+0000", 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.653792555999999, 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00025325, 2024-04-24T14:33:45.351 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684271+0000", 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684271+0000", 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684270+0000", 2024-04-24T14:33:45.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:45.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:45.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684288+0000", 2024-04-24T14:33:45.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7844000000000001e-05 2024-04-24T14:33:45.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:45.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684307+0000", 2024-04-24T14:33:45.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8759000000000001e-05 2024-04-24T14:33:45.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684315+0000", 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9200000000000004e-06 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684368+0000", 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3409000000000003e-05 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684524+0000", 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015631299999999999 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:118 3.2 e47/46 3:774f7988:ns::rbd_directory:head v 47'2, mlcod=0'0)", 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.686690+0000", 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.651373498, 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001318173, 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686690+0000", 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:45.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686690+0000", 2024-04-24T14:33:45.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:45.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686689+0000", 2024-04-24T14:33:45.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:45.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:45.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686698+0000", 2024-04-24T14:33:45.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8529999999999998e-06 2024-04-24T14:33:45.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686714+0000", 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5698000000000001e-05 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686721+0000", 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7719999999999997e-06 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686786+0000", 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5333000000000006e-05 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686818+0000", 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1998000000000001e-05 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.687962+0000", 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011436980000000001 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.688008+0000", 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6718999999999998e-05 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.358 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:119 3.2 e47/46 3:684f7435:::rbd_id.testrbdnbd16809:head v 47'3, mlcod=37'1)", 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.738094+0000", 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.599970015, 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016595679999999999, 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738094+0000", 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738094+0000", 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738092+0000", 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738116+0000", 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3883000000000002e-05 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738147+0000", 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1269000000000002e-05 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738159+0000", 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1443e-05 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738290+0000", 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013122399999999999 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738361+0000", 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1062000000000005e-05 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.739670+0000", 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013087540000000001 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.739753+0000", 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3607000000000001e-05 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.362 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:120 3.2 e47/46 3:5cea7035:::rbd_info:head v 47'4, mlcod=47'2)", 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.745883+0000", 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.592180703, 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0014160220000000001, 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745883+0000", 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745883+0000", 2024-04-24T14:33:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745881+0000", 2024-04-24T14:33:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745897+0000", 2024-04-24T14:33:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5519000000000002e-05 2024-04-24T14:33:45.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745952+0000", 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5071999999999997e-05 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745968+0000", 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6192999999999999e-05 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.746085+0000", 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117187 2024-04-24T14:33:45.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.746137+0000", 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2513999999999999e-05 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.747239+0000", 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00110176 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.747299+0000", 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9703000000000001e-05 2024-04-24T14:33:45.366 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.2661318880000003, 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:45.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:33:45.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:33:45.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:45.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:33:45.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:33:45.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:45.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:33:45.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:33:45.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:45.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:45.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:45.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:33:45.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:33:45.371 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.371 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.371 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.371 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:45.371 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:45.371 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:45.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:45.418 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:45.423+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:45.485 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:45.485 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:45.485 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:45.485 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:45.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:45.514 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:45.518+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:45.595 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:45.595 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:45.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:45.615 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:45.619+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:45.707 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.707 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:97 3.1 3:990e5750:::rbd_header.12ff64619e2d:head [watch reconnect cookie 140276853203408 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:33:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.3572706219999997, 2024-04-24T14:33:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012161632, 2024-04-24T14:33:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 97 2024-04-24T14:33:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:33:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:33:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352830+0000", 2024-04-24T14:33:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352834+0000", 2024-04-24T14:33:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2699999999999998e-06 2024-04-24T14:33:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352844+0000", 2024-04-24T14:33:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0406e-05 2024-04-24T14:33:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352850+0000", 2024-04-24T14:33:45.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6439999999999997e-06 2024-04-24T14:33:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362391+0000", 2024-04-24T14:33:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095408969999999996 2024-04-24T14:33:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362683+0000", 2024-04-24T14:33:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029238000000000002 2024-04-24T14:33:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.364992+0000", 2024-04-24T14:33:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023092020000000002 2024-04-24T14:33:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:225 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2048285099999996, 2024-04-24T14:33:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010959015000000001, 2024-04-24T14:33:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.712 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 225 2024-04-24T14:33:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505275+0000", 2024-04-24T14:33:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0269999999999998e-06 2024-04-24T14:33:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505279+0000", 2024-04-24T14:33:45.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4279999999999998e-06 2024-04-24T14:33:45.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505282+0000", 2024-04-24T14:33:45.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.982e-06 2024-04-24T14:33:45.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515209+0000", 2024-04-24T14:33:45.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099266419999999994 2024-04-24T14:33:45.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515466+0000", 2024-04-24T14:33:45.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025719700000000002 2024-04-24T14:33:45.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.516232+0000", 2024-04-24T14:33:45.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076608899999999996 2024-04-24T14:33:45.716 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.716 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.716 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.716 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:257 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:45.716 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.537079+0000", 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.1730224529999997, 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010535341, 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 257 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537079+0000", 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537079+0000", 2024-04-24T14:33:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537079+0000", 2024-04-24T14:33:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537081+0000", 2024-04-24T14:33:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9810000000000002e-06 2024-04-24T14:33:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537085+0000", 2024-04-24T14:33:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:33:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537088+0000", 2024-04-24T14:33:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1779999999999999e-06 2024-04-24T14:33:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.546841+0000", 2024-04-24T14:33:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097531560000000007 2024-04-24T14:33:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.546999+0000", 2024-04-24T14:33:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015763100000000001 2024-04-24T14:33:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.547614+0000", 2024-04-24T14:33:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061520699999999999 2024-04-24T14:33:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:273 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.553040+0000", 2024-04-24T14:33:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.1570612159999998, 2024-04-24T14:33:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010287627000000001, 2024-04-24T14:33:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 273 2024-04-24T14:33:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553040+0000", 2024-04-24T14:33:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553040+0000", 2024-04-24T14:33:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553039+0000", 2024-04-24T14:33:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553043+0000", 2024-04-24T14:33:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0690000000000002e-06 2024-04-24T14:33:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553049+0000", 2024-04-24T14:33:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3550000000000001e-06 2024-04-24T14:33:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553077+0000", 2024-04-24T14:33:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8065999999999999e-05 2024-04-24T14:33:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.562482+0000", 2024-04-24T14:33:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094054959999999993 2024-04-24T14:33:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.562646+0000", 2024-04-24T14:33:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163506 2024-04-24T14:33:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.563328+0000", 2024-04-24T14:33:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068183899999999995 2024-04-24T14:33:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:321 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.599796+0000", 2024-04-24T14:33:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.1103052409999998, 2024-04-24T14:33:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010461114000000001, 2024-04-24T14:33:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 321 2024-04-24T14:33:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599796+0000", 2024-04-24T14:33:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599796+0000", 2024-04-24T14:33:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599795+0000", 2024-04-24T14:33:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599799+0000", 2024-04-24T14:33:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.636e-06 2024-04-24T14:33:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599808+0000", 2024-04-24T14:33:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9199999999999993e-06 2024-04-24T14:33:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599817+0000", 2024-04-24T14:33:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0869999999999996e-06 2024-04-24T14:33:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.609395+0000", 2024-04-24T14:33:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009578105 2024-04-24T14:33:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.609583+0000", 2024-04-24T14:33:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018794300000000001 2024-04-24T14:33:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.610257+0000", 2024-04-24T14:33:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067424599999999998 2024-04-24T14:33:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:337 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.615565+0000", 2024-04-24T14:33:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.0945367150000003, 2024-04-24T14:33:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010598316999999999, 2024-04-24T14:33:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 337 2024-04-24T14:33:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615565+0000", 2024-04-24T14:33:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615565+0000", 2024-04-24T14:33:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615564+0000", 2024-04-24T14:33:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615568+0000", 2024-04-24T14:33:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.036e-06 2024-04-24T14:33:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615581+0000", 2024-04-24T14:33:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3317e-05 2024-04-24T14:33:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615587+0000", 2024-04-24T14:33:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5829999999999997e-06 2024-04-24T14:33:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.625169+0000", 2024-04-24T14:33:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095825949999999993 2024-04-24T14:33:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.625385+0000", 2024-04-24T14:33:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021516800000000001 2024-04-24T14:33:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.626163+0000", 2024-04-24T14:33:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077836799999999999 2024-04-24T14:33:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9537967680000001, 2024-04-24T14:33:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:33:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:33:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:33:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:33:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:33:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:33:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:33:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:33:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:33:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:33:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:33:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:33:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:33:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:33:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:33:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:33:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:33:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:33:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:33:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:33:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:33:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:33:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:33:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9537459340000001, 2024-04-24T14:33:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:33:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:33:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:33:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:33:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:33:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:33:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:33:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:33:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:33:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:33:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:33:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:33:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:33:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:33:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:33:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:33:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:33:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:33:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:33:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:33:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:33:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:33:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9537096269999998, 2024-04-24T14:33:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:33:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:33:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:33:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:33:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:33:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:33:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:33:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:33:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:33:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:33:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:33:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:33:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9502983770000002, 2024-04-24T14:33:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:33:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:33:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:33:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:33:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:33:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:33:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:33:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:33:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:33:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:33:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:33:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:33:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:33:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:33:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:33:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:33:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:33:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9502367, 2024-04-24T14:33:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:33:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:33:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:33:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:33:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:33:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:33:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:33:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:33:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:33:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:33:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:33:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:33:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:33:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:33:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:33:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:33:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.950202124, 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:33:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:33:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:33:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:33:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:33:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:33:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:33:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:33:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:33:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:33:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:33:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:33:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:33:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:33:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:33:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:33:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:33:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:33:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:33:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:33:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:33:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9501727849999999, 2024-04-24T14:33:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:33:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:33:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:33:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:33:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:33:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:33:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:33:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:33:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:33:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:33:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:33:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:33:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9500962030000002, 2024-04-24T14:33:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:33:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:33:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:33:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:33:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:33:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:33:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:33:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:33:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:33:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:33:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:33:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:33:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:33:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.950048658, 2024-04-24T14:33:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:33:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:33:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:33:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:33:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:33:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:33:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:33:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:33:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:33:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:33:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:33:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:33:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:33:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:33:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:33:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:33:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:33:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.950008016, 2024-04-24T14:33:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:33:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:33:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:33:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:33:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:33:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:33:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:33:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:33:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:33:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:33:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:33:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:33:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:33:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:33:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:33:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:33:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:33:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:33:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:33:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:33:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9499909049999999, 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:33:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:33:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:33:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:33:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:33:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:33:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:33:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:33:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:33:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:33:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:33:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:33:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:33:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:33:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:33:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:33:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:481 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:33:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9161191980000001, 2024-04-24T14:33:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010661718000000001, 2024-04-24T14:33:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 481 2024-04-24T14:33:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793981+0000", 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793985+0000", 2024-04-24T14:33:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8469999999999998e-06 2024-04-24T14:33:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:33:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0220000000000004e-06 2024-04-24T14:33:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793998+0000", 2024-04-24T14:33:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8559999999999998e-06 2024-04-24T14:33:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803807+0000", 2024-04-24T14:33:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098088629999999993 2024-04-24T14:33:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803916+0000", 2024-04-24T14:33:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010854899999999999 2024-04-24T14:33:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.804644+0000", 2024-04-24T14:33:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072835399999999996 2024-04-24T14:33:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:496 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.811896+0000", 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.8982058309999998, 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010339554000000001, 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 496 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811896+0000", 2024-04-24T14:33:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811896+0000", 2024-04-24T14:33:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811895+0000", 2024-04-24T14:33:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811898+0000", 2024-04-24T14:33:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1920000000000001e-06 2024-04-24T14:33:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811909+0000", 2024-04-24T14:33:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0722e-05 2024-04-24T14:33:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811914+0000", 2024-04-24T14:33:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6859999999999997e-06 2024-04-24T14:33:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.820742+0000", 2024-04-24T14:33:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0088287810000000008 2024-04-24T14:33:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.820912+0000", 2024-04-24T14:33:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169939 2024-04-24T14:33:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822235+0000", 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001322843 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:497 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.8981788860000002, 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013996471, 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 497 2024-04-24T14:33:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:33:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:33:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811922+0000", 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811925+0000", 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4120000000000001e-06 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811936+0000", 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1077000000000001e-05 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811941+0000", 2024-04-24T14:33:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7539999999999997e-06 2024-04-24T14:33:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822289+0000", 2024-04-24T14:33:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010347635000000001 2024-04-24T14:33:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822457+0000", 2024-04-24T14:33:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000168257 2024-04-24T14:33:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.825919+0000", 2024-04-24T14:33:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003461979 2024-04-24T14:33:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:45.811 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:45.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:45.813 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:45.814+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:45.913 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:45.918+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:45.916 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:45.916 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:45.916 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:45.916 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:45.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:46.015 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:46.018+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:46.023 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:46.023 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:46.032 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:46.116 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:46.119+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:46.134 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:46.134 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:189 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.468793+0000", 2024-04-24T14:33:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.6690373310000002, 2024-04-24T14:33:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099086950000000003, 2024-04-24T14:33:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 189 2024-04-24T14:33:46.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468793+0000", 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468793+0000", 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468792+0000", 2024-04-24T14:33:46.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468797+0000", 2024-04-24T14:33:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.189e-06 2024-04-24T14:33:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468825+0000", 2024-04-24T14:33:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8272999999999998e-05 2024-04-24T14:33:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468831+0000", 2024-04-24T14:33:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2430000000000002e-06 2024-04-24T14:33:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.477936+0000", 2024-04-24T14:33:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009104773 2024-04-24T14:33:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.478052+0000", 2024-04-24T14:33:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116131 2024-04-24T14:33:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.478702+0000", 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065004899999999998 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:192 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.6667307779999998, 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010036329, 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192 2024-04-24T14:33:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471099+0000", 2024-04-24T14:33:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471103+0000", 2024-04-24T14:33:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8800000000000001e-06 2024-04-24T14:33:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471111+0000", 2024-04-24T14:33:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2930000000000001e-06 2024-04-24T14:33:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471116+0000", 2024-04-24T14:33:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1800000000000004e-06 2024-04-24T14:33:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480338+0000", 2024-04-24T14:33:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092213069999999998 2024-04-24T14:33:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480467+0000", 2024-04-24T14:33:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012901500000000001 2024-04-24T14:33:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481136+0000", 2024-04-24T14:33:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066948400000000003 2024-04-24T14:33:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.6667077419999998, 2024-04-24T14:33:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:33:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:33:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:33:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:33:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:33:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:33:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:33:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2913510390000003, 2024-04-24T14:33:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:33:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:33:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:33:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:33:46.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:33:46.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:33:46.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:33:46.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:33:46.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:33:46.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:526 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2899635160000003, 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010568783999999999, 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.152 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 526 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847866+0000", 2024-04-24T14:33:46.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:46.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847871+0000", 2024-04-24T14:33:46.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1590000000000001e-06 2024-04-24T14:33:46.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847883+0000", 2024-04-24T14:33:46.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.172e-05 2024-04-24T14:33:46.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847890+0000", 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6950000000000003e-06 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857418+0000", 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095285909999999995 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857588+0000", 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169541 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858436+0000", 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084814700000000005 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2899228850000002, 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.156 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:33:46.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2898957959999997, 2024-04-24T14:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:33:46.160 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:33:46.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:33:46.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.164 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.289871829, 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:46.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2758878500000002, 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:46.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:33:46.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.275771539, 2024-04-24T14:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:33:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2757238810000002, 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:33:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:33:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:33:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2716865830000001, 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:33:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:33:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:33:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2716228239999996, 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:33:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:33:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.271597334, 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.201 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2715735840000004, 2024-04-24T14:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:33:46.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:33:46.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:33:46.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:33:46.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:33:46.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2715350709999997, 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:33:46.207 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:46.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:33:46.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:46.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.271506874, 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2714854170000001, 2024-04-24T14:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:33:46.218 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:33:46.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:33:46.223 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2714665749999998, 2024-04-24T14:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.224 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:33:46.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:33:46.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:561 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.2295039550000002, 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01066806, 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 561 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908326+0000", 2024-04-24T14:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908329+0000", 2024-04-24T14:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.802e-06 2024-04-24T14:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908338+0000", 2024-04-24T14:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6260000000000001e-06 2024-04-24T14:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908345+0000", 2024-04-24T14:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1389999999999998e-06 2024-04-24T14:33:46.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.917978+0000", 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096329840000000007 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918178+0000", 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019999199999999999 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918995+0000", 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081708099999999999 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.233 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.234 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.234 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:46.234 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:46.234 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:46.235 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:46.235 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:46.239+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:46.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:33:46.336 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:46.338+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:46.347 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:46.347 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:46.348 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:46.348 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:46.357 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:46.437 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:46.439+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:46.461 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:46.461 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:46.461 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:46.461 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:46.461 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:46.463 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:46.470 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:46.484 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:46.507 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:46.528 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:46.538 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:46.543+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:46.550 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:46.550 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:46.551 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:46.551 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:46.551 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:46.551 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:46.551 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:46.554 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:46.568 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:46.568 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:46.577 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:46.589 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd -p rbd --format xml du testrbdnbd16809 2024-04-24T14:33:46.589 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-04-24T14:33:46.638 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:46.638+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:46.664 INFO:tasks.workunit.client.0.smithi173.stderr:+ provisioned=67108864 2024-04-24T14:33:46.665 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd -p rbd --format xml du testrbdnbd16809 2024-04-24T14:33:46.665 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-04-24T14:33:46.681 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:46.681 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:46.690 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:46.739 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:46.743+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:46.742 INFO:tasks.workunit.client.0.smithi173.stderr:+ used=67108864 2024-04-24T14:33:46.742 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-04-24T14:33:46.743 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device /dev/nbd0 17315 2024-04-24T14:33:46.743 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=/dev/nbd0 2024-04-24T14:33:46.743 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=17315 2024-04-24T14:33:46.743 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:33:46.743 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:46.743 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:46.744 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:46.744 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:33:46.747 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:33:46.747 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:46.748 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:33:46.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T14:33:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:46.788 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:46.797 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:46.836 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:33:46.837 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^17315\b' 2024-04-24T14:33:46.837 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^17315\b' 2024-04-24T14:33:46.840 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:46.842+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:46.891 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:46.892 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:33:46.892 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *17315\b' 2024-04-24T14:33:46.892 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *17315\b' 2024-04-24T14:33:46.895 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:46.896 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:46.902 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:46.902 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:33:46.905 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:46.940 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:46.942+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:47.001 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:47.001 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:47.010 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:47.051 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:47.054+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:47.142 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:47.147+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:47.153 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:47.162 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:47.242 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:47.242+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:47.257 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:47.257 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:47.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:47.343 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:47.346+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:47.375 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:47.375 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:47.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:47.404 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:33:47.404 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:33:47.405 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:33:47.407 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:33:47.407 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:33:47.410 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:33:47.410 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:33:47.410 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:33:47.443 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:47.446+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:47.485 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:47.485 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:47.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:47.544 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:47.546+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:47.565 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:33:47.565 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:33:47.565 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:33:47.566 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:33:47.566 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:33:47.567 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:33:47.593 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:47.593 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:47.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:47.641 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=17473 2024-04-24T14:33:47.641 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 17473 2024-04-24T14:33:47.641 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 17473 -C rbd-nbd 2024-04-24T14:33:47.645 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:47.646+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:47.652 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:33:47.652 INFO:tasks.workunit.client.0.smithi173.stdout: 17473 ? 00:00:00 rbd-nbd 2024-04-24T14:33:47.653 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd -p rbd --format xml du testrbdnbd16809 2024-04-24T14:33:47.653 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-04-24T14:33:47.700 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:47.700 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:47.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:47.727 INFO:tasks.workunit.client.0.smithi173.stderr:+ provisioned=67108864 2024-04-24T14:33:47.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd -p rbd --format xml du testrbdnbd16809 2024-04-24T14:33:47.729 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-04-24T14:33:47.746 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:47.746+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:47.763 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-24T14:33:47.801 INFO:tasks.workunit.client.0.smithi173.stderr:+ used=67108864 2024-04-24T14:33:47.801 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-04-24T14:33:47.801 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo blkdiscard /dev/nbd0 2024-04-24T14:33:47.801 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:47.801 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:47.803 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:47.804 INFO:tasks.workunit.client.0.smithi173.stderr:++ which blkdiscard 2024-04-24T14:33:47.806 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T14:33:47.806 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:47.806 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-04-24T14:33:47.808 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:47.808 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:47.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:47.846 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:47.851+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:47.874 INFO:tasks.workunit.client.0.smithi173.stderr:+ sync 2024-04-24T14:33:47.914 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:47.914 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:47.923 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:47.967 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:47.970+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:48.027 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:48.027 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:48.037 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:48.049 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:48.051+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:48.053 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd -p rbd --format xml du testrbdnbd16809 2024-04-24T14:33:48.053 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-04-24T14:33:48.128 INFO:tasks.workunit.client.0.smithi173.stderr:+ provisioned=67108864 2024-04-24T14:33:48.129 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd -p rbd --format xml du testrbdnbd16809 2024-04-24T14:33:48.129 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-04-24T14:33:48.142 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:48.142 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:48.150 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:48.150+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:48.151 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:48.201 INFO:tasks.workunit.client.0.smithi173.stderr:+ used=0 2024-04-24T14:33:48.201 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 0 -lt 67108864 ']' 2024-04-24T14:33:48.201 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device /dev/nbd0 17473 2024-04-24T14:33:48.201 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=/dev/nbd0 2024-04-24T14:33:48.201 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=17473 2024-04-24T14:33:48.202 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:33:48.202 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:48.202 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:48.203 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:48.204 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:33:48.206 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:33:48.206 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:48.207 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:33:48.249 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:48.250 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:48.250 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:48.250+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:48.259 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:48.304 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:33:48.304 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^17473\b' 2024-04-24T14:33:48.304 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^17473\b' 2024-04-24T14:33:48.351 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:48.355+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:48.359 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:48.360 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:33:48.360 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *17473\b' 2024-04-24T14:33:48.360 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *17473\b' 2024-04-24T14:33:48.370 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:48.371 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:33:48.407 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:48.407 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:48.416 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:48.452 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:48.454+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:48.512 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:48.512 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:48.521 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:48.553 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:48.554+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:48.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:33:48.618 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:48.618 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:48.628 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:48.671 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:48.674+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:48.700 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:48.726 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:48.747 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:48.755 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:48.759+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:48.756 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [1] live_osds: [0, 2, 3, 4, 5] 2024-04-24T14:33:48.756 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:33:48.756 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4] 2024-04-24T14:33:48.756 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-04-24T14:33:48.770 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:48.776 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:48.776 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:48.786 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:48.805 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:48.806 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:48.806 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:48.806 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:48.806 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:48.806 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:48.806 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:48.809 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:48.810 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:48.855 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:48.858+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:48.873 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device -t nbd -o try-netlink map rbd/testrbdnbd16809 2024-04-24T14:33:48.873 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:33:48.873 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:33:48.875 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:33:48.876 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:33:48.878 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:33:48.879 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:33:48.879 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device -t nbd -o try-netlink map rbd/testrbdnbd16809 2024-04-24T14:33:48.887 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:48.887 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:48.896 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:48.956 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:48.958+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:49.036 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:33:49.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:33:49.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:33:49.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:33:49.037 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:33:49.037 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:33:49.044 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:1 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 35.848189241999997, 2024-04-24T14:33:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002159392, 2024-04-24T14:33:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 1 2024-04-24T14:33:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201710+0000", 2024-04-24T14:33:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:33:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201731+0000", 2024-04-24T14:33:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0835999999999998e-05 2024-04-24T14:33:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201769+0000", 2024-04-24T14:33:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7472999999999998e-05 2024-04-24T14:33:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201796+0000", 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6798999999999999e-05 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201923+0000", 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012770699999999999 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203873+0000", 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00194992 2024-04-24T14:33:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:1 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 35.846623504999997, 2024-04-24T14:33:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026872300000000001, 2024-04-24T14:33:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203279+0000", 2024-04-24T14:33:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203277+0000", 2024-04-24T14:33:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203292+0000", 2024-04-24T14:33:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4277000000000001e-05 2024-04-24T14:33:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203327+0000", 2024-04-24T14:33:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5219000000000003e-05 2024-04-24T14:33:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203350+0000", 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2807e-05 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203437+0000", 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7070999999999994e-05 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203548+0000", 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000111576 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:2 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 35.846510199999997, 2024-04-24T14:33:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047788300000000002, 2024-04-24T14:33:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203393+0000", 2024-04-24T14:33:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203391+0000", 2024-04-24T14:33:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203400+0000", 2024-04-24T14:33:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.5860000000000005e-06 2024-04-24T14:33:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203421+0000", 2024-04-24T14:33:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1056e-05 2024-04-24T14:33:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203434+0000", 2024-04-24T14:33:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3093e-05 2024-04-24T14:33:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203623+0000", 2024-04-24T14:33:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000188646 2024-04-24T14:33:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203871+0000", 2024-04-24T14:33:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00024812199999999998 2024-04-24T14:33:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:3 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 35.846452794999998, 2024-04-24T14:33:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059468100000000001, 2024-04-24T14:33:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203450+0000", 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203449+0000", 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203469+0000", 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0003000000000001e-05 2024-04-24T14:33:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203486+0000", 2024-04-24T14:33:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7309999999999999e-05 2024-04-24T14:33:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203501+0000", 2024-04-24T14:33:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.49e-05 2024-04-24T14:33:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203923+0000", 2024-04-24T14:33:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042195600000000002 2024-04-24T14:33:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204045+0000", 2024-04-24T14:33:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012202200000000001 2024-04-24T14:33:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:3 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 35.845081866999998, 2024-04-24T14:33:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00093837399999999996, 2024-04-24T14:33:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 3 2024-04-24T14:33:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204821+0000", 2024-04-24T14:33:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204819+0000", 2024-04-24T14:33:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204833+0000", 2024-04-24T14:33:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3346999999999999e-05 2024-04-24T14:33:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204866+0000", 2024-04-24T14:33:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3302000000000001e-05 2024-04-24T14:33:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.204903+0000", 2024-04-24T14:33:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7576000000000001e-05 2024-04-24T14:33:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205040+0000", 2024-04-24T14:33:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013616 2024-04-24T14:33:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205231+0000", 2024-04-24T14:33:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019118999999999999 2024-04-24T14:33:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205610+0000", 2024-04-24T14:33:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00037947000000000002 2024-04-24T14:33:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.205760+0000", 2024-04-24T14:33:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000149259 2024-04-24T14:33:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e47/46 cost 1048576)", 2024-04-24T14:33:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.684271+0000", 2024-04-24T14:33:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.365631926999999, 2024-04-24T14:33:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00025325, 2024-04-24T14:33:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684271+0000", 2024-04-24T14:33:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684271+0000", 2024-04-24T14:33:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684270+0000", 2024-04-24T14:33:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684288+0000", 2024-04-24T14:33:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7844000000000001e-05 2024-04-24T14:33:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684307+0000", 2024-04-24T14:33:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8759000000000001e-05 2024-04-24T14:33:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684315+0000", 2024-04-24T14:33:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9200000000000004e-06 2024-04-24T14:33:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684368+0000", 2024-04-24T14:33:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3409000000000003e-05 2024-04-24T14:33:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.684524+0000", 2024-04-24T14:33:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015631299999999999 2024-04-24T14:33:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:118 3.2 e47/46 3:774f7988:ns::rbd_directory:head v 47'2, mlcod=0'0)", 2024-04-24T14:33:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.686690+0000", 2024-04-24T14:33:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.363212869000002, 2024-04-24T14:33:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001318173, 2024-04-24T14:33:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686690+0000", 2024-04-24T14:33:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686690+0000", 2024-04-24T14:33:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686689+0000", 2024-04-24T14:33:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686698+0000", 2024-04-24T14:33:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.8529999999999998e-06 2024-04-24T14:33:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686714+0000", 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5698000000000001e-05 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686721+0000", 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7719999999999997e-06 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686786+0000", 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5333000000000006e-05 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.686818+0000", 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1998000000000001e-05 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.687962+0000", 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011436980000000001 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.688008+0000", 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6718999999999998e-05 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:119 3.2 e47/46 3:684f7435:::rbd_id.testrbdnbd16809:head v 47'3, mlcod=37'1)", 2024-04-24T14:33:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.738094+0000", 2024-04-24T14:33:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.311809386, 2024-04-24T14:33:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016595679999999999, 2024-04-24T14:33:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738094+0000", 2024-04-24T14:33:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738094+0000", 2024-04-24T14:33:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738092+0000", 2024-04-24T14:33:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738116+0000", 2024-04-24T14:33:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3883000000000002e-05 2024-04-24T14:33:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738147+0000", 2024-04-24T14:33:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1269000000000002e-05 2024-04-24T14:33:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738159+0000", 2024-04-24T14:33:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1443e-05 2024-04-24T14:33:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738290+0000", 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013122399999999999 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.738361+0000", 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1062000000000005e-05 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.739670+0000", 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013087540000000001 2024-04-24T14:33:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.739753+0000", 2024-04-24T14:33:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3607000000000001e-05 2024-04-24T14:33:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:120 3.2 e47/46 3:5cea7035:::rbd_info:head v 47'4, mlcod=47'2)", 2024-04-24T14:33:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.745883+0000", 2024-04-24T14:33:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.304020074, 2024-04-24T14:33:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0014160220000000001, 2024-04-24T14:33:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745883+0000", 2024-04-24T14:33:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745883+0000", 2024-04-24T14:33:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745881+0000", 2024-04-24T14:33:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745897+0000", 2024-04-24T14:33:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5519000000000002e-05 2024-04-24T14:33:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745952+0000", 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5071999999999997e-05 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.745968+0000", 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6192999999999999e-05 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.746085+0000", 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117187 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.746137+0000", 2024-04-24T14:33:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2513999999999999e-05 2024-04-24T14:33:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.747239+0000", 2024-04-24T14:33:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00110176 2024-04-24T14:33:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.747299+0000", 2024-04-24T14:33:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9703000000000001e-05 2024-04-24T14:33:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.9779712590000003, 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:33:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:33:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:33:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:33:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:33:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:33:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:33:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:33:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:33:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:33:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:33:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:33:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:33:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:33:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:33:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:49.087 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:49.088 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-04-24T14:33:49.067+0000 7ff9c25de700 -1 mon.a@0(leader).osd e57 definitely_dead 0 2024-04-24T14:33:49.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:49.090 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:49.091+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:49.104 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=17631 2024-04-24T14:33:49.104 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 17631 2024-04-24T14:33:49.104 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 17631 -C rbd-nbd 2024-04-24T14:33:49.115 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:33:49.115 INFO:tasks.workunit.client.0.smithi173.stdout: 17631 ? 00:00:00 rbd-nbd 2024-04-24T14:33:49.116 INFO:tasks.workunit.client.0.smithi173.stderr:++ basename /dev/nbd0 2024-04-24T14:33:49.117 INFO:tasks.workunit.client.0.smithi173.stderr:+ devname=nbd0 2024-04-24T14:33:49.118 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-04-24T14:33:49.121 INFO:tasks.workunit.client.0.smithi173.stderr:+ blocks=65536 2024-04-24T14:33:49.121 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 65536 2024-04-24T14:33:49.122 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd resize rbd/testrbdnbd16809 --size 128M 2024-04-24T14:33:49.190 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:49.190+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:49.193 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5169.0:20 3.1 3:990e5750:::rbd_header.12ff64619e2d:head [call rbd.set_size in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-04-24T14:33:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:49.197683+0000", 2024-04-24T14:33:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0021697600000000002, 2024-04-24T14:33:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022292029999999999, 2024-04-24T14:33:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2024-04-24T14:33:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5169", 2024-04-24T14:33:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3541535466", 2024-04-24T14:33:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:33:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:49.197683+0000", 2024-04-24T14:33:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:49.197683+0000", 2024-04-24T14:33:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:49.197682+0000", 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:49.197688+0000", 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5710000000000003e-06 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:49.197706+0000", 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8005000000000001e-05 2024-04-24T14:33:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:49.197713+0000", 2024-04-24T14:33:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.951e-06 2024-04-24T14:33:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:49.197822+0000", 2024-04-24T14:33:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108973 2024-04-24T14:33:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:49.198137+0000", 2024-04-24T14:33:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031461 2024-04-24T14:33:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:33:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:49.198695+0000", 2024-04-24T14:33:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055756600000000001 2024-04-24T14:33:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:33:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:49.199896+0000", 2024-04-24T14:33:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001201733 2024-04-24T14:33:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:33:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2024-04-24T14:33:49.198 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:49.198 INFO:tasks.workunit.client.0.smithi173.stderr: Resizing image: 100% complete...done. 2024-04-24T14:33:49.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:49.213 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd info rbd/testrbdnbd16809 2024-04-24T14:33:49.273 INFO:tasks.workunit.client.0.smithi173.stdout:rbd image 'testrbdnbd16809': 2024-04-24T14:33:49.273 INFO:tasks.workunit.client.0.smithi173.stdout: size 128 MiB in 32 objects 2024-04-24T14:33:49.274 INFO:tasks.workunit.client.0.smithi173.stdout: order 22 (4 MiB objects) 2024-04-24T14:33:49.274 INFO:tasks.workunit.client.0.smithi173.stdout: snapshot_count: 0 2024-04-24T14:33:49.274 INFO:tasks.workunit.client.0.smithi173.stdout: id: 12ff64619e2d 2024-04-24T14:33:49.274 INFO:tasks.workunit.client.0.smithi173.stdout: block_name_prefix: rbd_data.12ff64619e2d 2024-04-24T14:33:49.274 INFO:tasks.workunit.client.0.smithi173.stdout: format: 2 2024-04-24T14:33:49.274 INFO:tasks.workunit.client.0.smithi173.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-04-24T14:33:49.274 INFO:tasks.workunit.client.0.smithi173.stdout: op_features: 2024-04-24T14:33:49.274 INFO:tasks.workunit.client.0.smithi173.stdout: flags: 2024-04-24T14:33:49.274 INFO:tasks.workunit.client.0.smithi173.stdout: create_timestamp: Wed Apr 24 14:33:34 2024 2024-04-24T14:33:49.274 INFO:tasks.workunit.client.0.smithi173.stdout: access_timestamp: Wed Apr 24 14:33:34 2024 2024-04-24T14:33:49.274 INFO:tasks.workunit.client.0.smithi173.stdout: modify_timestamp: Wed Apr 24 14:33:34 2024 2024-04-24T14:33:49.280 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-04-24T14:33:49.284 INFO:tasks.workunit.client.0.smithi173.stderr:+ blocks2=131072 2024-04-24T14:33:49.284 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 131072 2024-04-24T14:33:49.284 INFO:tasks.workunit.client.0.smithi173.stderr:+ test 131072 -eq 131072 2024-04-24T14:33:49.284 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd resize rbd/testrbdnbd16809 --allow-shrink --size 64M 2024-04-24T14:33:49.290 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:49.290+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:49.295 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:49.295 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:49.295 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:49.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:49.358 INFO:tasks.workunit.client.0.smithi173.stderr: Resizing image: 50% complete... Resizing image: 53% complete... Resizing image: 56% complete... Resizing image: 59% complete... Resizing image: 62% complete... Resizing image: 65% complete... Resizing image: 68% complete... Resizing image: 71% complete... Resizing image: 75% complete... Resizing image: 78% complete... Resizing image: 81% complete... Resizing image: 84% complete... Resizing image: 87% complete... Resizing image: 90% complete... Resizing image: 93% complete... Resizing image: 96% complete... Resizing image: 100% complete...done. 2024-04-24T14:33:49.374 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-04-24T14:33:49.378 INFO:tasks.workunit.client.0.smithi173.stderr:+ blocks2=65536 2024-04-24T14:33:49.378 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 65536 2024-04-24T14:33:49.378 INFO:tasks.workunit.client.0.smithi173.stderr:+ test 65536 -eq 65536 2024-04-24T14:33:49.378 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device /dev/nbd0 17631 2024-04-24T14:33:49.378 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=/dev/nbd0 2024-04-24T14:33:49.378 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=17631 2024-04-24T14:33:49.378 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:33:49.378 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:49.379 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:49.380 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:49.381 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:33:49.383 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:33:49.384 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:49.384 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:33:49.391 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:49.395+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:49.409 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:49.409 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:49.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:49.409 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:49.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.409 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:97 3.1 3:990e5750:::rbd_header.12ff64619e2d:head [watch reconnect cookie 140276853203408 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:49.410 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:33:49.410 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0594998520000001, 2024-04-24T14:33:49.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012161632, 2024-04-24T14:33:49.410 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.410 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.410 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.410 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.410 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.410 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 97 2024-04-24T14:33:49.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.410 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352830+0000", 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352834+0000", 2024-04-24T14:33:49.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2699999999999998e-06 2024-04-24T14:33:49.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352844+0000", 2024-04-24T14:33:49.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0406e-05 2024-04-24T14:33:49.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352850+0000", 2024-04-24T14:33:49.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6439999999999997e-06 2024-04-24T14:33:49.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362391+0000", 2024-04-24T14:33:49.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095408969999999996 2024-04-24T14:33:49.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362683+0000", 2024-04-24T14:33:49.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029238000000000002 2024-04-24T14:33:49.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.364992+0000", 2024-04-24T14:33:49.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023092020000000002 2024-04-24T14:33:49.414 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.414 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.414 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.414 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:225 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:49.414 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:49.414 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.9070577399999999, 2024-04-24T14:33:49.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010959015000000001, 2024-04-24T14:33:49.414 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.414 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.414 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.415 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.415 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.415 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 225 2024-04-24T14:33:49.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.415 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:49.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:49.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505275+0000", 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0269999999999998e-06 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505279+0000", 2024-04-24T14:33:49.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4279999999999998e-06 2024-04-24T14:33:49.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505282+0000", 2024-04-24T14:33:49.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.982e-06 2024-04-24T14:33:49.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515209+0000", 2024-04-24T14:33:49.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099266419999999994 2024-04-24T14:33:49.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515466+0000", 2024-04-24T14:33:49.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025719700000000002 2024-04-24T14:33:49.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.516232+0000", 2024-04-24T14:33:49.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076608899999999996 2024-04-24T14:33:49.418 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.418 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.418 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:257 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.537079+0000", 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.8752516830000001, 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010535341, 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 257 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537079+0000", 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537079+0000", 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537079+0000", 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537081+0000", 2024-04-24T14:33:49.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9810000000000002e-06 2024-04-24T14:33:49.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537085+0000", 2024-04-24T14:33:49.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:33:49.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537088+0000", 2024-04-24T14:33:49.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1779999999999999e-06 2024-04-24T14:33:49.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.546841+0000", 2024-04-24T14:33:49.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097531560000000007 2024-04-24T14:33:49.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.546999+0000", 2024-04-24T14:33:49.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015763100000000001 2024-04-24T14:33:49.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.547614+0000", 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061520699999999999 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:273 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.553040+0000", 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.8592904460000002, 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010287627000000001, 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.424 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 273 2024-04-24T14:33:49.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.424 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553040+0000", 2024-04-24T14:33:49.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553040+0000", 2024-04-24T14:33:49.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553039+0000", 2024-04-24T14:33:49.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:49.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553043+0000", 2024-04-24T14:33:49.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0690000000000002e-06 2024-04-24T14:33:49.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553049+0000", 2024-04-24T14:33:49.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3550000000000001e-06 2024-04-24T14:33:49.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553077+0000", 2024-04-24T14:33:49.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8065999999999999e-05 2024-04-24T14:33:49.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.562482+0000", 2024-04-24T14:33:49.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094054959999999993 2024-04-24T14:33:49.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.562646+0000", 2024-04-24T14:33:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163506 2024-04-24T14:33:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.563328+0000", 2024-04-24T14:33:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068183899999999995 2024-04-24T14:33:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.428 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:321 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:49.428 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.599796+0000", 2024-04-24T14:33:49.428 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.8125344710000002, 2024-04-24T14:33:49.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010461114000000001, 2024-04-24T14:33:49.428 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.428 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.428 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.428 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.428 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.428 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 321 2024-04-24T14:33:49.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599796+0000", 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599796+0000", 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599795+0000", 2024-04-24T14:33:49.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599799+0000", 2024-04-24T14:33:49.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.636e-06 2024-04-24T14:33:49.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599808+0000", 2024-04-24T14:33:49.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9199999999999993e-06 2024-04-24T14:33:49.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599817+0000", 2024-04-24T14:33:49.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0869999999999996e-06 2024-04-24T14:33:49.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.609395+0000", 2024-04-24T14:33:49.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009578105 2024-04-24T14:33:49.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.609583+0000", 2024-04-24T14:33:49.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018794300000000001 2024-04-24T14:33:49.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.610257+0000", 2024-04-24T14:33:49.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067424599999999998 2024-04-24T14:33:49.432 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.432 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.432 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.432 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:337 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:49.432 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.615565+0000", 2024-04-24T14:33:49.432 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.7967659449999998, 2024-04-24T14:33:49.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010598316999999999, 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 337 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615565+0000", 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615565+0000", 2024-04-24T14:33:49.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615564+0000", 2024-04-24T14:33:49.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615568+0000", 2024-04-24T14:33:49.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.036e-06 2024-04-24T14:33:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615581+0000", 2024-04-24T14:33:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3317e-05 2024-04-24T14:33:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615587+0000", 2024-04-24T14:33:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5829999999999997e-06 2024-04-24T14:33:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.625169+0000", 2024-04-24T14:33:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095825949999999993 2024-04-24T14:33:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.625385+0000", 2024-04-24T14:33:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021516800000000001 2024-04-24T14:33:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.626163+0000", 2024-04-24T14:33:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077836799999999999 2024-04-24T14:33:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.6560259980000005, 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:49.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:49.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:33:49.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:49.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:33:49.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:33:49.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:33:49.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:33:49.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:33:49.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:33:49.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:33:49.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:33:49.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:49.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:33:49.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:33:49.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.655975164, 2024-04-24T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.6559388569999998, 2024-04-24T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.6525276069999997, 2024-04-24T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.65246593, 2024-04-24T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.652431354, 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.6524020149999998, 2024-04-24T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.6523254329999997, 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.6522778880000004, 2024-04-24T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.6522372460000003, 2024-04-24T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.6522201350000003, 2024-04-24T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:481 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.618348428, 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010661718000000001, 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 481 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793981+0000", 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793985+0000", 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8469999999999998e-06 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0220000000000004e-06 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793998+0000", 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8559999999999998e-06 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803807+0000", 2024-04-24T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098088629999999993 2024-04-24T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803916+0000", 2024-04-24T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010854899999999999 2024-04-24T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.804644+0000", 2024-04-24T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072835399999999996 2024-04-24T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:496 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.811896+0000", 2024-04-24T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.6004350609999998, 2024-04-24T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010339554000000001, 2024-04-24T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 496 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811896+0000", 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811896+0000", 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811895+0000", 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811898+0000", 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1920000000000001e-06 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811909+0000", 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0722e-05 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811914+0000", 2024-04-24T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6859999999999997e-06 2024-04-24T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.820742+0000", 2024-04-24T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0088287810000000008 2024-04-24T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.820912+0000", 2024-04-24T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169939 2024-04-24T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822235+0000", 2024-04-24T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001322843 2024-04-24T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:497 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.6004081159999997, 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013996471, 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 497 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811922+0000", 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811925+0000", 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4120000000000001e-06 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811936+0000", 2024-04-24T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1077000000000001e-05 2024-04-24T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811941+0000", 2024-04-24T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7539999999999997e-06 2024-04-24T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822289+0000", 2024-04-24T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010347635000000001 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822457+0000", 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000168257 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.825919+0000", 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003461979 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.516 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:49.517 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:33:49.517 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^17631\b' 2024-04-24T14:33:49.517 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^17631\b' 2024-04-24T14:33:49.518 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:49.518 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:49.523+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:49.525 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:49.525 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:33:49.526 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *17631\b' 2024-04-24T14:33:49.526 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *17631\b' 2024-04-24T14:33:49.531 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:49.532 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:33:49.618 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:49.618+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:49.665 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:49.665 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:49.666 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:49.666 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:49.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:49.719 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:49.723+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:49.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T14:33:49.776 INFO:teuthology.orchestra.run.smithi090.stderr:marked out osd.4. 2024-04-24T14:33:49.780 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:49.780 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:49.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:49.834 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:49.839+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:49.921 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:49.922+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:49.952 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:189 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.468793+0000", 2024-04-24T14:33:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.4876940059999999, 2024-04-24T14:33:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099086950000000003, 2024-04-24T14:33:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.954 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 189 2024-04-24T14:33:49.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.954 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468793+0000", 2024-04-24T14:33:49.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468793+0000", 2024-04-24T14:33:49.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468792+0000", 2024-04-24T14:33:49.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468797+0000", 2024-04-24T14:33:49.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.189e-06 2024-04-24T14:33:49.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468825+0000", 2024-04-24T14:33:49.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8272999999999998e-05 2024-04-24T14:33:49.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468831+0000", 2024-04-24T14:33:49.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2430000000000002e-06 2024-04-24T14:33:49.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.477936+0000", 2024-04-24T14:33:49.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009104773 2024-04-24T14:33:49.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.478052+0000", 2024-04-24T14:33:49.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116131 2024-04-24T14:33:49.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.478702+0000", 2024-04-24T14:33:49.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065004899999999998 2024-04-24T14:33:49.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.957 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.958 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:192 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:49.958 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:49.958 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.4853874529999995, 2024-04-24T14:33:49.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010036329, 2024-04-24T14:33:49.958 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.958 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.958 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192 2024-04-24T14:33:49.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.958 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:49.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:49.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471099+0000", 2024-04-24T14:33:49.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471103+0000", 2024-04-24T14:33:49.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8800000000000001e-06 2024-04-24T14:33:49.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471111+0000", 2024-04-24T14:33:49.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2930000000000001e-06 2024-04-24T14:33:49.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471116+0000", 2024-04-24T14:33:49.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1800000000000004e-06 2024-04-24T14:33:49.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480338+0000", 2024-04-24T14:33:49.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092213069999999998 2024-04-24T14:33:49.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480467+0000", 2024-04-24T14:33:49.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012901500000000001 2024-04-24T14:33:49.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481136+0000", 2024-04-24T14:33:49.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066948400000000003 2024-04-24T14:33:49.962 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.962 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.962 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.962 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:49.962 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:49.962 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.4853644169999995, 2024-04-24T14:33:49.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:33:49.962 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.963 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.963 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:33:49.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.963 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:49.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:33:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:33:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:33:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:33:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:33:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:33:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:33:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:33:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:33:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:33:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:33:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:33:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:33:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.110007714, 2024-04-24T14:33:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:33:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:33:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:33:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:33:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:33:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:33:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:33:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:33:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:33:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:33:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:33:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:33:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:33:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:33:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:526 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.108620191, 2024-04-24T14:33:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010568783999999999, 2024-04-24T14:33:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 526 2024-04-24T14:33:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847866+0000", 2024-04-24T14:33:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847871+0000", 2024-04-24T14:33:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1590000000000001e-06 2024-04-24T14:33:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847883+0000", 2024-04-24T14:33:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.172e-05 2024-04-24T14:33:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847890+0000", 2024-04-24T14:33:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6950000000000003e-06 2024-04-24T14:33:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857418+0000", 2024-04-24T14:33:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095285909999999995 2024-04-24T14:33:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857588+0000", 2024-04-24T14:33:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169541 2024-04-24T14:33:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858436+0000", 2024-04-24T14:33:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084814700000000005 2024-04-24T14:33:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.1085795600000008, 2024-04-24T14:33:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:33:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:33:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:33:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:33:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:33:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:33:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:33:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:33:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:33:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:33:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:33:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:33:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:33:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:33:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:33:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.1085524709999994, 2024-04-24T14:33:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:33:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:33:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:33:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:33:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:33:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:33:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:33:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:33:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:33:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:33:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:33:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.1085285040000006, 2024-04-24T14:33:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:33:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:33:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:33:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:33:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:33:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:33:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:33:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:33:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:33:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:33:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:33:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:33:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:33:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:33:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:33:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0945445249999999, 2024-04-24T14:33:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:33:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:33:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:33:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:33:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:33:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:33:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:33:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:33:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:33:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:33:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:33:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:33:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:33:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:33:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:33:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:33:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0944282140000006, 2024-04-24T14:33:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:33:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:33:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:33:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:33:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:33:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:33:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:33:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:33:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:33:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:33:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:33:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:33:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:33:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:33:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:33:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:33:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:33:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:33:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:33:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0943805560000008, 2024-04-24T14:33:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:33:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:33:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:33:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:33:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:33:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:33:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:33:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:33:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:33:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:33:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:33:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:33:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:33:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:33:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:33:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:33:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:33:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0903432580000008, 2024-04-24T14:33:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:33:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:33:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:33:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:33:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:33:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:33:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:33:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:33:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:33:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:33:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:33:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:33:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:33:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:33:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:33:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:33:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:33:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:33:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:33:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0902794989999993, 2024-04-24T14:33:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:33:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:33:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:33:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:33:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:33:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:33:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:33:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:33:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:33:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:33:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:33:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:33:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:33:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:33:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:33:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:33:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:33:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:33:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:33:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:33:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:33:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:33:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:33:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0902540090000006, 2024-04-24T14:33:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:33:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:33:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:33:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:33:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:33:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:33:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:33:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:33:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:33:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:33:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:33:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:33:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:33:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:33:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:33:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:33:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:33:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:33:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:33:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:33:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:33:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0902302590000001, 2024-04-24T14:33:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:33:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:33:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:33:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:33:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:33:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:33:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:33:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:33:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:33:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:33:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:33:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:33:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:33:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:33:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:33:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:33:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:33:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0901917460000004, 2024-04-24T14:33:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:33:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:33:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:33:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:33:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:33:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:33:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:33:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:33:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:33:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:33:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:33:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:33:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:33:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:33:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:33:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:33:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:33:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:33:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:33:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0901635489999997, 2024-04-24T14:33:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:33:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:33:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:33:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:33:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:33:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:33:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:33:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:33:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:33:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:33:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:33:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:33:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:33:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:33:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:33:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:33:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:33:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:33:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:33:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:33:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0901420920000007, 2024-04-24T14:33:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:33:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:33:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:33:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:33:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:33:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:33:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:33:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:33:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:33:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:33:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:33:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:33:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:33:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:33:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:33:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:33:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:33:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:33:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:33:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:33:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0901232499999995, 2024-04-24T14:33:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:33:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:33:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:33:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:33:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:33:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:33:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:33:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:33:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:33:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:33:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:33:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:33:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:33:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:33:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:33:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:33:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:33:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:33:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:33:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:33:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:561 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:33:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0481606299999999, 2024-04-24T14:33:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01066806, 2024-04-24T14:33:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 561 2024-04-24T14:33:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:33:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:33:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908326+0000", 2024-04-24T14:33:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908329+0000", 2024-04-24T14:33:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.802e-06 2024-04-24T14:33:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908338+0000", 2024-04-24T14:33:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6260000000000001e-06 2024-04-24T14:33:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908345+0000", 2024-04-24T14:33:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1389999999999998e-06 2024-04-24T14:33:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.917978+0000", 2024-04-24T14:33:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096329840000000007 2024-04-24T14:33:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918178+0000", 2024-04-24T14:33:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019999199999999999 2024-04-24T14:33:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918995+0000", 2024-04-24T14:33:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081708099999999999 2024-04-24T14:33:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:50.059 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:50.061 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:50.061 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd --device-type nbd map --read-only rbd/testrbdnbd16809 2024-04-24T14:33:50.061 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:33:50.062 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:33:50.062 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:33:50.062 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:33:50.062 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:33:50.062 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:33:50.062 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --read-only rbd/testrbdnbd16809 2024-04-24T14:33:50.062 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:50.062+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:50.161 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:50.161 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:50.161 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:50.161 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:50.162 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:50.163+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:50.170 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:50.262 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:50.267+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:50.263 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:50.263 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:50.263 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:50.263 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:50.263 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:50.273 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:50.363 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:50.366+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:50.372 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 88.049664065000002, 2024-04-24T14:33:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:33:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:33:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:33:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:33:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:33:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:33:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:33:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:33:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:33:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:33:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:33:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:33:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:33:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:33:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:33:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:33:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:33:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:33:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:33:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:33:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:33:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:33:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.750149123, 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:33:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:33:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:33:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:33:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:33:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:33:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:33:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:33:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:33:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:33:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:33:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.749923354999993, 2024-04-24T14:33:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:33:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:33:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:33:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:33:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:33:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:33:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:33:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:33:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:33:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:33:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:33:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:33:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:33:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:33:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:33:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:33:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:33:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:33:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:33:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:33:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:33:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:33:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:33:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.749746267999996, 2024-04-24T14:33:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:33:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:33:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:33:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:33:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:33:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:33:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:33:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:33:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:33:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:33:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:33:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:33:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:33:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:33:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:33:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:33:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:33:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:33:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.749642354000002, 2024-04-24T14:33:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:33:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:33:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:33:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:33:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:33:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:33:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:33:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:33:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:33:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:33:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:33:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:33:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:33:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:33:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:33:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:33:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:33:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:33:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:33:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:33:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:33:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:33:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:33:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.749346033999998, 2024-04-24T14:33:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:33:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:33:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:33:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:33:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:33:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:33:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:33:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:33:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:33:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:33:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:33:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:33:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:33:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:33:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:33:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:33:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:33:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:33:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:33:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:33:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:33:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:33:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:33:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.749253902999996, 2024-04-24T14:33:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:33:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:33:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:33:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:33:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:33:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:33:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:33:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:33:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:33:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:33:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:33:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:33:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:33:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:33:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:33:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:33:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:33:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:33:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:33:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:33:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:33:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:33:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.749168690999994, 2024-04-24T14:33:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:33:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:33:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:33:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:33:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:33:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:33:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:33:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:33:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:33:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:33:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:33:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:33:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:33:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:33:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:33:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:33:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:33:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:33:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:33:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:33:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:33:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:33:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:33:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.748931810000002, 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:33:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:33:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:33:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:33:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:33:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:33:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:33:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:33:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:33:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:33:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:33:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:33:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:33:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:33:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:33:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:33:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:33:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:33:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:33:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:33:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.748850288, 2024-04-24T14:33:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:33:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:33:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:33:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:33:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:33:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:33:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:33:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:33:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:33:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:33:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:33:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:33:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:33:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:33:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:33:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:33:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:33:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:33:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:33:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:33:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:33:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:33:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:33:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.317091876999996, 2024-04-24T14:33:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:33:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:33:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:33:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:33:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:33:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:33:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:33:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:33:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:33:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:33:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:33:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:33:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:33:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:33:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:33:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:33:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:33:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:33:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:33:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:33:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:33:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:33:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:33:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.317050374000004, 2024-04-24T14:33:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:33:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:33:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:33:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:33:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:33:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:33:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:33:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:33:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:33:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:33:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:33:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:33:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:33:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:33:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:33:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:33:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:33:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:33:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:33:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:33:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:33:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:33:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.316937941000006, 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:33:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:33:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:33:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:33:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:33:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:33:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:33:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:33:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:33:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:33:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:33:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:33:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:33:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:33:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:33:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:33:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.316909640999995, 2024-04-24T14:33:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:33:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:33:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:33:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:33:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:33:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:33:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:33:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:33:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:33:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:33:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:33:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:33:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:33:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:33:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:33:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:33:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:33:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:33:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.316821937, 2024-04-24T14:33:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:33:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:33:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:33:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:33:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:33:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:33:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:33:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:33:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:33:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:33:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:33:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:33:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:33:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:33:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:33:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:33:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:33:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:33:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:33:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.316699135999997, 2024-04-24T14:33:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:33:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:33:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:33:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:33:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:33:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:33:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:33:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:33:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:33:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:33:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:33:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:33:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:33:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:33:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:33:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:33:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:33:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:33:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:33:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:33:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:33:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:33:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:33:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:33:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 86.673459203999997, 2024-04-24T14:33:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:33:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:33:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:33:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:33:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:33:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:33:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:33:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:33:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:33:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:33:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:33:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:33:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:33:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:33:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:33:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:33:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:33:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:33:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:33:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:33:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:33:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:33:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:33:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:33:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.010454576000001, 2024-04-24T14:33:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:33:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:33:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:33:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:33:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:33:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:33:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:33:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:33:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:33:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:33:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:33:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:33:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:33:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:33:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:33:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:33:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:33:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:33:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:33:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:33:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:33:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:33:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.316330577, 2024-04-24T14:33:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:33:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:33:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:33:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:33:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:33:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:33:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:33:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:33:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:33:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:33:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:33:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:33:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:33:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:33:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:33:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:33:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:33:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:33:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:33:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:33:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:577 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.923702+0000", 2024-04-24T14:33:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.4520248339999995, 2024-04-24T14:33:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012522472, 2024-04-24T14:33:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 577 2024-04-24T14:33:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923702+0000", 2024-04-24T14:33:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923702+0000", 2024-04-24T14:33:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923701+0000", 2024-04-24T14:33:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923708+0000", 2024-04-24T14:33:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.233e-06 2024-04-24T14:33:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923723+0000", 2024-04-24T14:33:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5099000000000001e-05 2024-04-24T14:33:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923733+0000", 2024-04-24T14:33:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:33:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.935366+0000", 2024-04-24T14:33:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011632105 2024-04-24T14:33:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.935446+0000", 2024-04-24T14:33:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012000000000006e-05 2024-04-24T14:33:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.936224+0000", 2024-04-24T14:33:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077877899999999995 2024-04-24T14:33:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.489 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.489 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.489 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:50.490 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:50.490 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:50.494+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:50.591 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:50.590+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:50.632 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:50.632 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:50.632 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:50.632 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:50.641 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:50.691 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:50.695+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:50.737 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:50.737 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:50.737 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:50.738 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:50.738 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:50.747 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:50.780 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-24T14:33:50.791 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:50.794+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:50.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:33:50.856 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:50.856 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:50.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:50.856 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:50.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.856 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:379 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.856 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:50.857 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1966043559999999, 2024-04-24T14:33:50.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068078410000000002, 2024-04-24T14:33:50.857 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.857 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.857 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.857 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.857 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.857 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 379 2024-04-24T14:33:50.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.857 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:50.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:50.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:50.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664240+0000", 2024-04-24T14:33:50.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6539999999999999e-06 2024-04-24T14:33:50.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664245+0000", 2024-04-24T14:33:50.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2179999999999998e-06 2024-04-24T14:33:50.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664249+0000", 2024-04-24T14:33:50.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2279999999999998e-06 2024-04-24T14:33:50.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670245+0000", 2024-04-24T14:33:50.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059962430000000001 2024-04-24T14:33:50.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670400+0000", 2024-04-24T14:33:50.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015535399999999999 2024-04-24T14:33:50.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671046+0000", 2024-04-24T14:33:50.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064591599999999996 2024-04-24T14:33:50.861 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.861 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.861 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.861 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:380 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.861 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:33:50.861 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1965577290000002, 2024-04-24T14:33:50.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076160830000000001, 2024-04-24T14:33:50.861 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.862 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.862 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.862 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.862 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.862 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 380 2024-04-24T14:33:50.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.862 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:33:50.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:33:50.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664284+0000", 2024-04-24T14:33:50.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664287+0000", 2024-04-24T14:33:50.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9739999999999998e-06 2024-04-24T14:33:50.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664302+0000", 2024-04-24T14:33:50.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5214999999999999e-05 2024-04-24T14:33:50.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664307+0000", 2024-04-24T14:33:50.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6859999999999997e-06 2024-04-24T14:33:50.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671098+0000", 2024-04-24T14:33:50.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067912479999999997 2024-04-24T14:33:50.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671253+0000", 2024-04-24T14:33:50.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015427400000000001 2024-04-24T14:33:50.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671901+0000", 2024-04-24T14:33:50.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064845600000000005 2024-04-24T14:33:50.865 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.865 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.865 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.866 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.866 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:50.866 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1965281839999999, 2024-04-24T14:33:50.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:33:50.866 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.866 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.866 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.866 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.866 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.866 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:33:50.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.867 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:50.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:50.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:33:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:33:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:33:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:33:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:33:50.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:33:50.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:33:50.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:33:50.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:33:50.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:33:50.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:33:50.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:33:50.870 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.870 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.870 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.870 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:382 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.870 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:33:50.870 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1935267009999997, 2024-04-24T14:33:50.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067029910000000002, 2024-04-24T14:33:50.871 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.871 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.871 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.871 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.871 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.871 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 382 2024-04-24T14:33:50.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.871 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:33:50.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:33:50.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667315+0000", 2024-04-24T14:33:50.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667321+0000", 2024-04-24T14:33:50.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2099999999999998e-06 2024-04-24T14:33:50.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667336+0000", 2024-04-24T14:33:50.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4612e-05 2024-04-24T14:33:50.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667343+0000", 2024-04-24T14:33:50.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0119999999999999e-06 2024-04-24T14:33:50.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672957+0000", 2024-04-24T14:33:50.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056139700000000002 2024-04-24T14:33:50.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.673127+0000", 2024-04-24T14:33:50.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017044599999999999 2024-04-24T14:33:50.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674019+0000", 2024-04-24T14:33:50.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00089165499999999999 2024-04-24T14:33:50.875 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.875 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.875 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.875 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.875 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:50.875 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1934936579999995, 2024-04-24T14:33:50.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:33:50.875 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.875 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.875 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.875 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.876 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.876 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:33:50.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.876 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:50.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:50.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:50.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:33:50.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:33:50.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:33:50.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:33:50.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:33:50.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:33:50.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:33:50.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:33:50.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:33:50.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:33:50.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:33:50.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:33:50.879 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.879 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.879 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.879 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.880 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:50.880 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1934694720000003, 2024-04-24T14:33:50.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:33:50.880 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.880 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.880 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.880 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.880 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.880 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:33:50.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.880 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:50.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:50.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:50.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:33:50.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:33:50.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:33:50.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:33:50.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:33:50.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:33:50.884 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.884 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.884 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.884 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.884 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:33:50.884 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1934520880000008, 2024-04-24T14:33:50.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:33:50.884 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.885 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.885 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.885 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.885 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.885 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:33:50.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.885 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:33:50.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:33:50.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:33:50.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:33:50.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:33:50.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:33:50.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:33:50.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:33:50.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:33:50.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:33:50.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:33:50.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:33:50.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:33:50.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:33:50.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:33:50.888 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.889 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.889 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.889 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:399 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.889 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:33:50.889 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1790419530000005, 2024-04-24T14:33:50.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071317810000000002, 2024-04-24T14:33:50.889 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.889 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.889 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.889 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.889 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.889 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 399 2024-04-24T14:33:50.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.890 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:33:50.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:33:50.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681800+0000", 2024-04-24T14:33:50.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681804+0000", 2024-04-24T14:33:50.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.715e-06 2024-04-24T14:33:50.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681816+0000", 2024-04-24T14:33:50.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1851000000000001e-05 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681820+0000", 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6450000000000004e-06 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688059+0000", 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062390090000000002 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688196+0000", 2024-04-24T14:33:50.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013721999999999999 2024-04-24T14:33:50.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688933+0000", 2024-04-24T14:33:50.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073605000000000003 2024-04-24T14:33:50.893 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.893 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.893 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.893 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.893 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:33:50.893 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1790159960000004, 2024-04-24T14:33:50.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:33:50.894 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.894 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.894 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.894 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.894 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.894 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:33:50.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.894 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:33:50.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:33:50.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:33:50.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:33:50.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:33:50.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:33:50.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:33:50.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:33:50.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:33:50.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:33:50.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:33:50.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:33:50.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:33:50.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:33:50.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:33:50.897 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.898 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.898 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.898 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.898 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:33:50.898 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1789836969999996, 2024-04-24T14:33:50.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:33:50.898 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.898 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.898 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.898 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.898 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.898 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:33:50.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.899 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:33:50.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:33:50.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:33:50.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:33:50.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:33:50.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:33:50.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:33:50.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:33:50.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:33:50.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:33:50.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:33:50.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:33:50.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:33:50.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:33:50.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:33:50.902 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.902 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.902 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.903 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:414 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.903 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:33:50.903 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1647882870000004, 2024-04-24T14:33:50.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069877910000000001, 2024-04-24T14:33:50.903 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.903 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.903 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.903 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.903 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.903 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 414 2024-04-24T14:33:50.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.903 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:33:50.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:33:50.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696053+0000", 2024-04-24T14:33:50.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696058+0000", 2024-04-24T14:33:50.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7509999999999999e-06 2024-04-24T14:33:50.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696074+0000", 2024-04-24T14:33:50.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5743000000000002e-05 2024-04-24T14:33:50.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696093+0000", 2024-04-24T14:33:50.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9508e-05 2024-04-24T14:33:50.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702314+0000", 2024-04-24T14:33:50.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062207010000000004 2024-04-24T14:33:50.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702431+0000", 2024-04-24T14:33:50.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011723299999999999 2024-04-24T14:33:50.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703042+0000", 2024-04-24T14:33:50.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000610752 2024-04-24T14:33:50.907 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.907 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.907 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.907 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.907 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:33:50.907 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1647314200000007, 2024-04-24T14:33:50.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:33:50.908 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.908 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.908 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.908 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.908 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.908 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:33:50.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.908 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:33:50.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:33:50.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:33:50.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:33:50.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:33:50.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:33:50.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:33:50.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:33:50.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:33:50.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:33:50.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:33:50.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:33:50.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:33:50.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:33:50.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:33:50.912 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.912 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.912 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.912 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.912 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:50.912 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.16470232, 2024-04-24T14:33:50.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:33:50.912 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.912 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.912 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.912 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.912 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.913 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:33:50.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.913 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:50.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:50.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:50.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:33:50.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:33:50.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:33:50.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:33:50.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:33:50.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:33:50.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:33:50.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:33:50.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:33:50.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:33:50.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:33:50.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:33:50.916 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.916 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.916 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.917 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.917 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:50.917 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1646763900000003, 2024-04-24T14:33:50.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:33:50.917 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.917 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.917 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.917 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.917 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.917 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:33:50.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.917 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:50.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:50.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:50.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:33:50.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:33:50.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:33:50.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:33:50.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:33:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:33:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:33:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:33:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:33:50.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:33:50.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:33:50.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:33:50.921 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.921 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.921 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.921 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:429 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3538944~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.921 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:33:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1503949500000008, 2024-04-24T14:33:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071866789999999996, 2024-04-24T14:33:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 429 2024-04-24T14:33:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:33:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:33:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710447+0000", 2024-04-24T14:33:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710451+0000", 2024-04-24T14:33:50.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5930000000000002e-06 2024-04-24T14:33:50.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710465+0000", 2024-04-24T14:33:50.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.411e-05 2024-04-24T14:33:50.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710470+0000", 2024-04-24T14:33:50.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3129999999999998e-06 2024-04-24T14:33:50.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716696+0000", 2024-04-24T14:33:50.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006225983 2024-04-24T14:33:50.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716901+0000", 2024-04-24T14:33:50.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020550200000000001 2024-04-24T14:33:50.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717634+0000", 2024-04-24T14:33:50.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073292099999999998 2024-04-24T14:33:50.926 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.926 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.926 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.926 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:430 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.926 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:33:50.926 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1490306419999996, 2024-04-24T14:33:50.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068326970000000004, 2024-04-24T14:33:50.926 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.926 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.926 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.926 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 430 2024-04-24T14:33:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:33:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:33:50.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711810+0000", 2024-04-24T14:33:50.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:50.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711822+0000", 2024-04-24T14:33:50.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2166e-05 2024-04-24T14:33:50.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711856+0000", 2024-04-24T14:33:50.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3816000000000003e-05 2024-04-24T14:33:50.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711873+0000", 2024-04-24T14:33:50.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6742999999999999e-05 2024-04-24T14:33:50.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717693+0000", 2024-04-24T14:33:50.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058204579999999997 2024-04-24T14:33:50.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717864+0000", 2024-04-24T14:33:50.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017128299999999999 2024-04-24T14:33:50.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718645+0000", 2024-04-24T14:33:50.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078033699999999998 2024-04-24T14:33:50.930 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.930 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.930 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.931 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.931 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:33:50.931 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1489559259999993, 2024-04-24T14:33:50.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:33:50.931 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.931 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:50.931 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.931 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.931 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.931 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:33:50.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.931 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:33:50.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:33:50.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:33:50.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:33:50.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:33:50.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:33:50.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:33:50.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:33:50.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:33:50.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:33:50.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:33:50.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:33:50.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:33:50.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:33:50.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:33:50.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.935 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.935 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.935 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:33:50.935 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1489239940000004, 2024-04-24T14:33:50.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:33:50.935 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.935 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:50.935 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.936 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.936 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.936 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:33:50.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.936 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:33:50.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:33:50.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:33:50.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:50.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:33:50.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:33:50.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:33:50.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:33:50.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:33:50.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:33:50.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:33:50.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:33:50.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:33:50.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:33:50.939 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.939 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.939 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.939 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:50.939 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:33:50.939 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.1488564780000008, 2024-04-24T14:33:50.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:33:50.939 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:33:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:33:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:33:50.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:33:50.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:50.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:33:50.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:33:50.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:33:50.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:33:50.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:33:50.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:33:50.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:33:50.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:33:50.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:33:50.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:33:50.943 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.943 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.943 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.943 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:33:50.943 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:50.943 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.0914435949999994, 2024-04-24T14:33:50.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:33:50.944 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:50.944 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:50.944 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:50.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:50.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:50.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:50.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:50.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:50.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:50.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:50.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:50.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:50.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:33:50.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:33:50.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:50.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:33:50.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:33:50.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:50.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:33:50.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:33:50.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:50.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:33:50.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:33:50.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:50.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:33:50.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:33:50.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:50.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:33:50.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:33:50.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:50.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:50.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:50.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:33:50.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:33:50.948 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.948 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.948 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.948 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:50.948 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:50.948 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:50.949 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:50.950 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:50.950+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:50.978 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:51.000 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:51.023 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:51.046 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:51.049 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:51.050+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:51.053 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:51.054 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:51.054 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:51.054 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:51.063 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:51.070 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:51.151 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:51.155+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:51.158 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:51.159 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:51.167 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:51.251 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:51.255+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:51.260 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:51.260 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:51.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:51.352 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:51.354+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:51.377 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:51.377 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:51.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:51.454 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:51.454+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:51.487 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:51.487 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:51.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:51.554 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:51.559+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:51.596 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:51.596 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:51.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:51.655 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:51.659+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:51.747 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:51.748 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:51.756 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:51.762+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:51.757 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:51.822 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:33:51.823 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd list 2024-04-24T14:33:51.823 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk -v pool=rbd -v img=testrbdnbd16809 -v dev=/dev/nbd0 '$2 == pool && $3 == img && $5 == dev {print $1}' 2024-04-24T14:33:51.857 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:51.859+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:51.863 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:51.863 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:51.872 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:51.893 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=17759 2024-04-24T14:33:51.893 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 17759 2024-04-24T14:33:51.893 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 17759 -C rbd-nbd 2024-04-24T14:33:51.901 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:33:51.901 INFO:tasks.workunit.client.0.smithi173.stdout: 17759 ? 00:00:00 rbd-nbd 2024-04-24T14:33:51.901 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo dd if=/dev/nbd0 of=/dev/null bs=1M 2024-04-24T14:33:51.901 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:51.902 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:51.903 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:51.904 INFO:tasks.workunit.client.0.smithi173.stderr:++ which dd 2024-04-24T14:33:51.905 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/dd 2024-04-24T14:33:51.905 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:51.906 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/dd if=/dev/nbd0 of=/dev/null bs=1M 2024-04-24T14:33:51.958 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:51.958+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:51.977 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:51.977 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:51.987 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:52.047 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records in 2024-04-24T14:33:52.047 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records out 2024-04-24T14:33:52.048 INFO:tasks.workunit.client.0.smithi173.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.136202 s, 493 MB/s 2024-04-24T14:33:52.049 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false _sudo dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M oflag=direct 2024-04-24T14:33:52.049 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M oflag=direct 2024-04-24T14:33:52.049 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:52.050 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:52.052 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:52.052 INFO:tasks.workunit.client.0.smithi173.stderr:++ which dd 2024-04-24T14:33:52.055 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/dd 2024-04-24T14:33:52.055 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:52.055 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M oflag=direct 2024-04-24T14:33:52.059 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:52.063+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:52.065 INFO:tasks.workunit.client.0.smithi173.stderr:/usr/bin/dd: error writing '/dev/nbd0': Operation not permitted 2024-04-24T14:33:52.065 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records in 2024-04-24T14:33:52.065 INFO:tasks.workunit.client.0.smithi173.stderr:0+0 records out 2024-04-24T14:33:52.065 INFO:tasks.workunit.client.0.smithi173.stderr:0 bytes copied, 0.0011061 s, 0.0 kB/s 2024-04-24T14:33:52.066 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:52.066 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device /dev/nbd0 17759 2024-04-24T14:33:52.066 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=/dev/nbd0 2024-04-24T14:33:52.067 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=17759 2024-04-24T14:33:52.067 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:33:52.067 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:52.067 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:52.069 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:52.070 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:33:52.072 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:33:52.072 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:52.072 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:33:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:52.093 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:52.103 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:52.160 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:52.162+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:52.162 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:33:52.162 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^17759\b' 2024-04-24T14:33:52.162 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^17759\b' 2024-04-24T14:33:52.207 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:52.207 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:52.211 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:52.212 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *17759\b' 2024-04-24T14:33:52.212 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:33:52.212 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *17759\b' 2024-04-24T14:33:52.216 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:52.226 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:52.226 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:33:52.260 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:52.262+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:52.313 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:52.313 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:52.322 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:52.367 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:52.370+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:52.462 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:52.462+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:52.470 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:52.479 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:52.563 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:52.567+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:52.578 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:52.587 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:52.664 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:52.671+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:52.684 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:52.685 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:52.694 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:52.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd --device-type nbd map --exclusive rbd/testrbdnbd16809 2024-04-24T14:33:52.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:33:52.729 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:33:52.731 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:33:52.731 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:33:52.734 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:33:52.734 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:33:52.734 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --exclusive rbd/testrbdnbd16809 2024-04-24T14:33:52.765 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:52.771+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:52.793 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:52.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T14:33:52.807 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:52.866 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:52.866+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:52.903 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:52.912 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:52.966 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:52.971+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:53.007 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:53.008 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:53.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:53.067 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:53.071+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:53.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:33:53.116 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:53.116 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:53.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:53.168 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:53.171+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:53.198 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:53.215 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:53.237 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:53.258 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:53.262 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:53.262 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:53.270 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:53.275+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:53.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:53.278 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:53.278 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:53.279 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:53.279 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:53.279 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:53.279 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:53.279 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:53.281 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:53.281 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:53.371 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:53.374+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:53.371 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:53.371 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:53.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:97 3.1 3:990e5750:::rbd_header.12ff64619e2d:head [watch reconnect cookie 140276853203408 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 12.021772821000001, 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012161632, 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 97 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352830+0000", 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352834+0000", 2024-04-24T14:33:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2699999999999998e-06 2024-04-24T14:33:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352844+0000", 2024-04-24T14:33:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0406e-05 2024-04-24T14:33:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352850+0000", 2024-04-24T14:33:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6439999999999997e-06 2024-04-24T14:33:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362391+0000", 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095408969999999996 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362683+0000", 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029238000000000002 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.364992+0000", 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023092020000000002 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.375 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:225 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.869330709, 2024-04-24T14:33:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010959015000000001, 2024-04-24T14:33:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 225 2024-04-24T14:33:53.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:53.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505275+0000", 2024-04-24T14:33:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0269999999999998e-06 2024-04-24T14:33:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505279+0000", 2024-04-24T14:33:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4279999999999998e-06 2024-04-24T14:33:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505282+0000", 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.982e-06 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515209+0000", 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099266419999999994 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515466+0000", 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025719700000000002 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.516232+0000", 2024-04-24T14:33:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076608899999999996 2024-04-24T14:33:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:257 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.537079+0000", 2024-04-24T14:33:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.837524652000001, 2024-04-24T14:33:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010535341, 2024-04-24T14:33:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.380 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 257 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537079+0000", 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537079+0000", 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537079+0000", 2024-04-24T14:33:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537081+0000", 2024-04-24T14:33:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9810000000000002e-06 2024-04-24T14:33:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537085+0000", 2024-04-24T14:33:53.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.537088+0000", 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1779999999999999e-06 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.546841+0000", 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0097531560000000007 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.546999+0000", 2024-04-24T14:33:53.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015763100000000001 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.547614+0000", 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061520699999999999 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:273 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.553040+0000", 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.821563415, 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010287627000000001, 2024-04-24T14:33:53.384 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 273 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553040+0000", 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553040+0000", 2024-04-24T14:33:53.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553039+0000", 2024-04-24T14:33:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553043+0000", 2024-04-24T14:33:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0690000000000002e-06 2024-04-24T14:33:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553049+0000", 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3550000000000001e-06 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.553077+0000", 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8065999999999999e-05 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.562482+0000", 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094054959999999993 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.562646+0000", 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163506 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.563328+0000", 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068183899999999995 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:321 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.599796+0000", 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.77480744, 2024-04-24T14:33:53.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010461114000000001, 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 321 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599796+0000", 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599796+0000", 2024-04-24T14:33:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599795+0000", 2024-04-24T14:33:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599799+0000", 2024-04-24T14:33:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.636e-06 2024-04-24T14:33:53.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599808+0000", 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9199999999999993e-06 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.599817+0000", 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0869999999999996e-06 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.609395+0000", 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009578105 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.609583+0000", 2024-04-24T14:33:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018794300000000001 2024-04-24T14:33:53.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.610257+0000", 2024-04-24T14:33:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067424599999999998 2024-04-24T14:33:53.393 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:337 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.615565+0000", 2024-04-24T14:33:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.759038914, 2024-04-24T14:33:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010598316999999999, 2024-04-24T14:33:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.394 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 337 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615565+0000", 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615565+0000", 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615564+0000", 2024-04-24T14:33:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615568+0000", 2024-04-24T14:33:53.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.036e-06 2024-04-24T14:33:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615581+0000", 2024-04-24T14:33:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3317e-05 2024-04-24T14:33:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.615587+0000", 2024-04-24T14:33:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5829999999999997e-06 2024-04-24T14:33:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.625169+0000", 2024-04-24T14:33:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095825949999999993 2024-04-24T14:33:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.625385+0000", 2024-04-24T14:33:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021516800000000001 2024-04-24T14:33:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.626163+0000", 2024-04-24T14:33:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077836799999999999 2024-04-24T14:33:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.398 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.618298966999999, 2024-04-24T14:33:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:33:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:33:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.399 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:33:53.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:33:53.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:33:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:33:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:33:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:33:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:33:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:33:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:33:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:33:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:33:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:53.404 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.618248133, 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:33:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:33:53.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:33:53.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:33:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:33:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:33:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:33:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:33:53.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:33:53.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:33:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.618211826, 2024-04-24T14:33:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:33:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.410 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:33:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:33:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:33:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:33:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:33:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:33:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:33:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:33:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:33:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:33:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:33:53.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:33:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:33:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:33:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:33:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.614800576, 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:33:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:33:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:33:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:33:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:33:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:33:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:33:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:33:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:33:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:33:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:33:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:33:53.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:33:53.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:33:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:33:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.614738899000001, 2024-04-24T14:33:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:33:53.421 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:33:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:33:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:33:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:33:53.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:33:53.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:33:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:33:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:33:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:33:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:33:53.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:33:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:33:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:33:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:33:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:33:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:33:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:33:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.614704323, 2024-04-24T14:33:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:33:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.427 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:33:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:33:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:33:53.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:33:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:33:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:33:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:33:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:33:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:33:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:33:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:33:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:33:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:33:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:33:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.432 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.614674984000001, 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:33:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:33:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:33:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:33:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:33:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:33:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:33:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:33:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:33:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:33:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:33:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:33:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:33:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:33:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:33:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:33:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:33:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:33:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:33:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:33:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:33:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.614598402, 2024-04-24T14:33:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:33:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:33:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:33:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:33:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:33:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:33:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:33:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:33:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:33:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:33:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:33:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:33:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:33:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:33:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:33:53.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:33:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:33:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:33:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:33:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:33:53.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:33:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:33:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:33:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.614550856999999, 2024-04-24T14:33:53.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:33:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:33:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:33:53.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:33:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:33:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:33:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:33:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:33:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:33:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:33:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:33:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:33:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:33:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:33:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:33:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:33:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.614510214999999, 2024-04-24T14:33:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:33:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.450 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:33:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:53.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:33:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:33:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:33:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:33:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:33:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:33:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:33:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:33:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:33:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:33:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:33:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:33:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:33:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:33:53.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:33:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:33:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:33:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:33:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:33:53.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:33:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.614493103999999, 2024-04-24T14:33:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:33:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.456 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:33:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:33:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:33:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:33:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:33:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:33:53.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:33:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:33:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:33:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:33:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:33:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:33:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:33:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:33:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:33:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:33:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.461 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:481 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:33:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.580621397, 2024-04-24T14:33:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010661718000000001, 2024-04-24T14:33:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 481 2024-04-24T14:33:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:33:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:33:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793981+0000", 2024-04-24T14:33:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793985+0000", 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8469999999999998e-06 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0220000000000004e-06 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793998+0000", 2024-04-24T14:33:53.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8559999999999998e-06 2024-04-24T14:33:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803807+0000", 2024-04-24T14:33:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098088629999999993 2024-04-24T14:33:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803916+0000", 2024-04-24T14:33:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010854899999999999 2024-04-24T14:33:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.804644+0000", 2024-04-24T14:33:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072835399999999996 2024-04-24T14:33:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:496 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.811896+0000", 2024-04-24T14:33:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.56270803, 2024-04-24T14:33:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010339554000000001, 2024-04-24T14:33:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.466 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 496 2024-04-24T14:33:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811896+0000", 2024-04-24T14:33:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811896+0000", 2024-04-24T14:33:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811895+0000", 2024-04-24T14:33:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811898+0000", 2024-04-24T14:33:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1920000000000001e-06 2024-04-24T14:33:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811909+0000", 2024-04-24T14:33:53.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0722e-05 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811914+0000", 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6859999999999997e-06 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.820742+0000", 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0088287810000000008 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.820912+0000", 2024-04-24T14:33:53.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169939 2024-04-24T14:33:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822235+0000", 2024-04-24T14:33:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001322843 2024-04-24T14:33:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:497 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:33:53.470 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.562681084999999, 2024-04-24T14:33:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013996471, 2024-04-24T14:33:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 497 2024-04-24T14:33:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:33:53.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:33:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811922+0000", 2024-04-24T14:33:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811925+0000", 2024-04-24T14:33:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4120000000000001e-06 2024-04-24T14:33:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811936+0000", 2024-04-24T14:33:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1077000000000001e-05 2024-04-24T14:33:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811941+0000", 2024-04-24T14:33:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7539999999999997e-06 2024-04-24T14:33:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822289+0000", 2024-04-24T14:33:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010347635000000001 2024-04-24T14:33:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822457+0000", 2024-04-24T14:33:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000168257 2024-04-24T14:33:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.825919+0000", 2024-04-24T14:33:53.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003461979 2024-04-24T14:33:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.475 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.475 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:53.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:53.477 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:53.482+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:53.577 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:53.578+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:53.579 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:53.579 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:53.579 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:53.579 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:53.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:53.678 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:53.682+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:53.687 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:53.687 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:53.687 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:53.687 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:53.687 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:53.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:53.779 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:53.783+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:53.802 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:53.802 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:53.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:53.802 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:53.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.802 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:189 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:53.802 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.468793+0000", 2024-04-24T14:33:53.802 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 12.336292319, 2024-04-24T14:33:53.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099086950000000003, 2024-04-24T14:33:53.803 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.803 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.803 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 189 2024-04-24T14:33:53.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.803 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468793+0000", 2024-04-24T14:33:53.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468793+0000", 2024-04-24T14:33:53.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468792+0000", 2024-04-24T14:33:53.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468797+0000", 2024-04-24T14:33:53.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.189e-06 2024-04-24T14:33:53.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468825+0000", 2024-04-24T14:33:53.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8272999999999998e-05 2024-04-24T14:33:53.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.468831+0000", 2024-04-24T14:33:53.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2430000000000002e-06 2024-04-24T14:33:53.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.477936+0000", 2024-04-24T14:33:53.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009104773 2024-04-24T14:33:53.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.478052+0000", 2024-04-24T14:33:53.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116131 2024-04-24T14:33:53.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.478702+0000", 2024-04-24T14:33:53.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065004899999999998 2024-04-24T14:33:53.807 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.807 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.807 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.808 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:192 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:53.808 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:53.808 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 12.333985766, 2024-04-24T14:33:53.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010036329, 2024-04-24T14:33:53.808 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.808 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.808 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.808 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.808 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.808 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192 2024-04-24T14:33:53.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.809 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:53.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:53.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471099+0000", 2024-04-24T14:33:53.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471103+0000", 2024-04-24T14:33:53.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8800000000000001e-06 2024-04-24T14:33:53.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471111+0000", 2024-04-24T14:33:53.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2930000000000001e-06 2024-04-24T14:33:53.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471116+0000", 2024-04-24T14:33:53.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1800000000000004e-06 2024-04-24T14:33:53.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480338+0000", 2024-04-24T14:33:53.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092213069999999998 2024-04-24T14:33:53.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480467+0000", 2024-04-24T14:33:53.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012901500000000001 2024-04-24T14:33:53.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481136+0000", 2024-04-24T14:33:53.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066948400000000003 2024-04-24T14:33:53.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.813 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.813 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:53.813 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:53.813 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 12.33396273, 2024-04-24T14:33:53.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:33:53.813 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.814 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.814 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:33:53.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.814 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:53.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:53.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:33:53.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:33:53.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:33:53.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:33:53.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:33:53.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:33:53.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:33:53.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:33:53.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:33:53.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:33:53.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:33:53.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:33:53.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:33:53.818 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.818 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.818 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.819 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.819 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:53.819 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.958606027, 2024-04-24T14:33:53.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:33:53.819 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.819 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.819 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.819 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.819 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.820 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:33:53.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.820 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:53.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:53.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:33:53.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:33:53.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:33:53.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:33:53.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:33:53.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:33:53.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:33:53.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:33:53.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:33:53.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:33:53.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:33:53.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:33:53.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:33:53.824 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.824 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.824 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.824 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:526 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.824 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:53.824 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.957218504, 2024-04-24T14:33:53.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010568783999999999, 2024-04-24T14:33:53.825 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.825 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.825 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.825 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.825 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.825 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 526 2024-04-24T14:33:53.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.825 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:53.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:53.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847866+0000", 2024-04-24T14:33:53.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847871+0000", 2024-04-24T14:33:53.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1590000000000001e-06 2024-04-24T14:33:53.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847883+0000", 2024-04-24T14:33:53.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.172e-05 2024-04-24T14:33:53.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847890+0000", 2024-04-24T14:33:53.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6950000000000003e-06 2024-04-24T14:33:53.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857418+0000", 2024-04-24T14:33:53.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095285909999999995 2024-04-24T14:33:53.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857588+0000", 2024-04-24T14:33:53.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169541 2024-04-24T14:33:53.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858436+0000", 2024-04-24T14:33:53.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084814700000000005 2024-04-24T14:33:53.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.829 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.830 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.830 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:53.830 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.957177872999999, 2024-04-24T14:33:53.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:33:53.830 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.830 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.830 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:33:53.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.831 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:53.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:53.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:33:53.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:33:53.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:33:53.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:33:53.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:33:53.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:33:53.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:33:53.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:33:53.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:33:53.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:33:53.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:33:53.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:33:53.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:33:53.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.835 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.835 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.835 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:53.835 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.957150784, 2024-04-24T14:33:53.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:33:53.835 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.835 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.835 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.836 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:33:53.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.836 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:53.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:53.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:33:53.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:33:53.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:33:53.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:33:53.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:33:53.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:33:53.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:33:53.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:33:53.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:33:53.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:33:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:33:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:33:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:33:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.957126817000001, 2024-04-24T14:33:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:33:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:33:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.841 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:53.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:53.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:33:53.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:33:53.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:33:53.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:33:53.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:33:53.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:33:53.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:33:53.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:33:53.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:33:53.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:33:53.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:33:53.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:33:53.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:33:53.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.845 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.846 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.846 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.846 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:53.846 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.943142838, 2024-04-24T14:33:53.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:33:53.846 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.846 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.847 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:33:53.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.847 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:53.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:53.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:33:53.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:53.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:33:53.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:33:53.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:33:53.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:33:53.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:33:53.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:33:53.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:33:53.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:33:53.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:33:53.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:33:53.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:33:53.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:33:53.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:33:53.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:33:53.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:33:53.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:33:53.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:33:53.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:33:53.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:33:53.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:33:53.852 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.852 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.853 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.853 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:53.853 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.943026527000001, 2024-04-24T14:33:53.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:33:53.853 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.853 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.854 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:33:53.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.854 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:53.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:53.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:33:53.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:53.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:33:53.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:33:53.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:33:53.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:33:53.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:33:53.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:33:53.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:33:53.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:33:53.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:33:53.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:33:53.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:33:53.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:33:53.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:33:53.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:33:53.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:33:53.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:33:53.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:33:53.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:33:53.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:33:53.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:33:53.860 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.860 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.860 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.860 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.860 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:53.860 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.942978869000001, 2024-04-24T14:33:53.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:33:53.860 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.861 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.861 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.861 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.861 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.861 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:33:53.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.861 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:53.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:53.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:33:53.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:33:53.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:33:53.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:33:53.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:33:53.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:33:53.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:33:53.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:33:53.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:33:53.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:33:53.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:33:53.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:33:53.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:33:53.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:33:53.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:33:53.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:33:53.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:33:53.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:33:53.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:33:53.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:33:53.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:33:53.867 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.867 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.867 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.867 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.867 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:53.867 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.938941571000001, 2024-04-24T14:33:53.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:33:53.868 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.868 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.868 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.868 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.868 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.868 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:33:53.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.868 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:53.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:53.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:33:53.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:53.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:33:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:33:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:33:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:33:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:33:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:33:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:33:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:33:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:33:53.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:33:53.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:33:53.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:33:53.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:33:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:33:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:33:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:33:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:33:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:33:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:33:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:33:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.938877811999999, 2024-04-24T14:33:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:33:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:33:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:33:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:33:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:33:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:33:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:33:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:33:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:33:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:33:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:33:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:33:53.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:33:53.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:33:53.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:33:53.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:33:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:33:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:33:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:33:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:33:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:33:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:33:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:33:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.938852322000001, 2024-04-24T14:33:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:33:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:33:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:33:53.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:33:53.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:33:53.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:33:53.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:33:53.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:33:53.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:33:53.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:33:53.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:33:53.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:33:53.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:33:53.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:33:53.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:33:53.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:33:53.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:33:53.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:33:53.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:33:53.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:33:53.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:33:53.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:33:53.888 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.888 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.888 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.888 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.888 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:53.888 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.938828572, 2024-04-24T14:33:53.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:33:53.888 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:33:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:33:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:33:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:33:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:33:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:33:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:33:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:33:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:33:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:33:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:33:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:33:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:33:53.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:33:53.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:33:53.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:33:53.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:33:53.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:33:53.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:33:53.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:33:53.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:33:53.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:33:53.894 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.895 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.895 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.895 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:53.895 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.938790059, 2024-04-24T14:33:53.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:33:53.895 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.895 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.895 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.895 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:33:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:33:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:33:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:33:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:33:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:33:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:33:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:33:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:33:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:33:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:33:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:33:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:33:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:33:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:33:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:33:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:33:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:33:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:33:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:33:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:33:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:33:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:33:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.938761862, 2024-04-24T14:33:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:33:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:33:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:33:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:33:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:33:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:33:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:33:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:33:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:33:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:33:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:33:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:33:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:33:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:33:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:33:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:33:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:33:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:33:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:33:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:33:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:33:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:33:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:33:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:33:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:33:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:33:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.938740405000001, 2024-04-24T14:33:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:33:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:33:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:33:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:33:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:33:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:33:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:33:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:33:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:33:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:33:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:33:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:33:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:33:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:33:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:33:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:33:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:33:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:33:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:33:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:33:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:33:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:33:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:33:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:33:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:33:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:33:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.938721563, 2024-04-24T14:33:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:33:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:33:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:33:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:33:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:33:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:33:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:33:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:33:53.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:33:53.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:33:53.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:33:53.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:33:53.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:33:53.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:53.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:33:53.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:33:53.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:33:53.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:33:53.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:53.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:33:53.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:33:53.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:33:53.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:33:53.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:33:53.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:33:53.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:33:53.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:33:53.922 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.922 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.922 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.922 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:561 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:53.923 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:33:53.923 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.896758943, 2024-04-24T14:33:53.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01066806, 2024-04-24T14:33:53.923 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:53.923 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:53.923 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:53.923 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:53.923 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:53.923 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 561 2024-04-24T14:33:53.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.923 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:53.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:53.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:33:53.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:53.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:33:53.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:53.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:53.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908326+0000", 2024-04-24T14:33:53.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:53.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:53.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908329+0000", 2024-04-24T14:33:53.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.802e-06 2024-04-24T14:33:53.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:53.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908338+0000", 2024-04-24T14:33:53.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6260000000000001e-06 2024-04-24T14:33:53.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:53.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908345+0000", 2024-04-24T14:33:53.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1389999999999998e-06 2024-04-24T14:33:53.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:53.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.917978+0000", 2024-04-24T14:33:53.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096329840000000007 2024-04-24T14:33:53.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:53.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918178+0000", 2024-04-24T14:33:53.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019999199999999999 2024-04-24T14:33:53.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:53.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:53.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:53.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918995+0000", 2024-04-24T14:33:53.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081708099999999999 2024-04-24T14:33:53.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.927 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:53.927 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:53.928 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:53.929 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-24T14:33:53.930 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:33:53.930 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:33:53.930 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:33:53.930 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:33:53.930 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:33:53.930 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:33:53.931 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:53.932 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:53.930+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:53.933 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=17833 2024-04-24T14:33:53.933 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 17833 2024-04-24T14:33:53.933 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 17833 -C rbd-nbd 2024-04-24T14:33:53.941 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:33:53.941 INFO:tasks.workunit.client.0.smithi173.stdout: 17833 ? 00:00:00 rbd-nbd 2024-04-24T14:33:53.941 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M oflag=direct 2024-04-24T14:33:53.941 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:53.941 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:53.943 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:53.943 INFO:tasks.workunit.client.0.smithi173.stderr:++ which dd 2024-04-24T14:33:53.946 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/dd 2024-04-24T14:33:53.946 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:53.946 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M oflag=direct 2024-04-24T14:33:54.029 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:54.031+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:54.041 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:54.042 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:54.042 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:54.042 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:54.052 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:54.130 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:54.130+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:54.150 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:54.151 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:54.160 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:54.232 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:54.234+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:54.260 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:54.260 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:54.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:54.260 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:54.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.260 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.260 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:54.260 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.937028744000003, 2024-04-24T14:33:54.261 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:33:54.261 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.261 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.261 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.261 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.261 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.261 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:33:54.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.261 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.261 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.261 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:54.261 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.262 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:54.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.262 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:33:54.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:54.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.262 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:33:54.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:33:54.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.263 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.263 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:33:54.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:33:54.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.263 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.263 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:33:54.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:33:54.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:33:54.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:33:54.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:33:54.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:33:54.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:33:54.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:33:54.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:33:54.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:33:54.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:33:54.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:33:54.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:33:54.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:33:54.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.266 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.266 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:33:54.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:33:54.266 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.266 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.266 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.266 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.266 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:54.266 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.637513802000001, 2024-04-24T14:33:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:33:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:33:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.267 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:54.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:33:54.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:54.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:33:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:33:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:33:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:33:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:33:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:33:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:33:54.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:33:54.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:33:54.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:33:54.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:33:54.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:33:54.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:33:54.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:33:54.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:33:54.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:33:54.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:33:54.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:33:54.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.637288033999994, 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.272 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:33:54.274 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:54.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:33:54.274 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:33:54.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:33:54.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:33:54.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:33:54.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:33:54.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:33:54.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:33:54.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.276 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.276 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:33:54.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:33:54.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.276 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.276 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:33:54.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:33:54.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.276 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.276 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:33:54.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:33:54.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:33:54.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:33:54.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:33:54.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:33:54.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.277 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.278 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:33:54.278 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:33:54.278 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.278 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.278 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.278 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.278 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.278 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.278 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:54.278 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.637110946999996, 2024-04-24T14:33:54.278 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:33:54.278 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.278 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.279 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.279 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.279 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.279 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:33:54.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.279 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:54.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:54.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:33:54.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:54.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:33:54.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:33:54.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:33:54.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:33:54.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.281 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:33:54.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:33:54.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.281 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:33:54.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:33:54.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:33:54.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:33:54.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:33:54.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:33:54.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:33:54.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:33:54.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.283 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.283 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:33:54.283 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:33:54.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.283 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.283 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:33:54.283 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:33:54.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.283 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.283 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.637007033000003, 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.284 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:33:54.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:54.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.286 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.286 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:33:54.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:33:54.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.286 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.286 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:33:54.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:33:54.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:33:54.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:33:54.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:33:54.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:33:54.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:33:54.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:33:54.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:33:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:33:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:33:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:33:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:33:54.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:33:54.289 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.289 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.289 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:33:54.289 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:33:54.289 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.289 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.289 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:33:54.289 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:33:54.289 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.289 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.289 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.290 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.290 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.290 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:54.290 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.636710712999999, 2024-04-24T14:33:54.290 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:33:54.290 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.290 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.290 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.290 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.290 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.290 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:33:54.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:33:54.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:54.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:33:54.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:33:54.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:33:54.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:33:54.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:33:54.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:33:54.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:33:54.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:33:54.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:33:54.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:33:54.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.294 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.294 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:33:54.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:33:54.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.294 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.294 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:33:54.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:33:54.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.294 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.294 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:33:54.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:33:54.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:33:54.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:33:54.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:33:54.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:33:54.295 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.295 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.295 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.296 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.296 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:54.296 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.636618581999997, 2024-04-24T14:33:54.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:33:54.296 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.296 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.296 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.296 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.296 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.296 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:33:54.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.296 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.296 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:33:54.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:33:54.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:33:54.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:33:54.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:33:54.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:33:54.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:33:54.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:33:54.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:33:54.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:33:54.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:33:54.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:33:54.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:33:54.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.301 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.301 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:33:54.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:33:54.301 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.301 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.301 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.301 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.301 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:54.301 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.636533369999995, 2024-04-24T14:33:54.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:33:54.301 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.302 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.302 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.302 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.302 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.302 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:33:54.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.302 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:54.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:33:54.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:33:54.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:33:54.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:33:54.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:33:54.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:33:54.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:33:54.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:33:54.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:33:54.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:33:54.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:33:54.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:33:54.306 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.636296489000003, 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:33:54.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.308 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:54.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:54.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:33:54.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:33:54.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:33:54.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:33:54.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:33:54.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:33:54.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:54.312 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.636214967000001, 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:54.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:33:54.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:54.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:33:54.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:33:54.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:33:54.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:33:54.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:33:54.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:33:54.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:33:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:33:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.204456555999997, 2024-04-24T14:33:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:33:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.318 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:33:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:33:54.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:33:54.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:33:54.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:33:54.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:33:54.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:33:54.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:33:54.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.324 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.324 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:54.324 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.204415053000005, 2024-04-24T14:33:54.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:33:54.324 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.324 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.324 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.324 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.324 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.324 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:33:54.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.324 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:33:54.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:33:54.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:33:54.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:33:54.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:33:54.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:33:54.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:33:54.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:33:54.329 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.329 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.329 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.329 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.329 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:54.329 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.204302619999993, 2024-04-24T14:33:54.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:33:54.329 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.329 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.330 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.330 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.330 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.330 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:33:54.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.330 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:54.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:33:54.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:33:54.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:33:54.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:33:54.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:33:54.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:33:54.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:33:54.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:33:54.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:33:54.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:33:54.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:33:54.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:33:54.334 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.334 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.204274319999996, 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.335 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:54.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:54.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:33:54.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:33:54.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:33:54.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:33:54.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:33:54.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:33:54.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:33:54.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:33:54.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:33:54.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:33:54.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:33:54.340 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.340 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.340 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.340 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.340 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:54.341 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.204186616000001, 2024-04-24T14:33:54.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:33:54.341 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.341 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.341 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.341 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.341 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.341 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:33:54.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.341 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:54.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:54.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:33:54.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:54.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:33:54.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:33:54.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:33:54.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:33:54.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:33:54.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:33:54.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:33:54.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:33:54.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:33:54.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:33:54.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:33:54.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:33:54.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:33:54.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:33:54.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:33:54.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:33:54.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:33:54.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:33:54.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:33:54.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:33:54.346 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.346 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.346 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.346 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:54.346 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:54.346 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.204063814999998, 2024-04-24T14:33:54.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:33:54.346 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:33:54.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:33:54.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:33:54.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:33:54.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:33:54.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:33:54.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:33:54.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:33:54.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:33:54.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:33:54.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:33:54.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:33:54.351 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.351 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 90.560823882999998, 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:54.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:54.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:33:54.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:54.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:33:54.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:33:54.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:33:54.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:33:54.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:33:54.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:33:54.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:33:54.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:33:54.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:33:54.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:33:54.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:54.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:33:54.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:33:54.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:54.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:33:54.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:33:54.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:54.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:33:54.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:33:54.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:33:54.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:33:54.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:33:54.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:33:54.357 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.357 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.358 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.358 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:33:54.358 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:54.358 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.897819255000002, 2024-04-24T14:33:54.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:33:54.358 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.358 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.358 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.358 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:33:54.359 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:33:54.359 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:33:54.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.359 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:54.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:54.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:33:54.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:54.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:33:54.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:33:54.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:33:54.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:33:54.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:33:54.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:33:54.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:33:54.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:33:54.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:33:54.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:33:54.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:33:54.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:33:54.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:33:54.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:33:54.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:33:54.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:33:54.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:33:54.363 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.363 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.363 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.363 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.363 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:54.363 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 14.203695256, 2024-04-24T14:33:54.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:33:54.364 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.364 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.364 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.364 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:33:54.364 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:33:54.364 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:33:54.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.364 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:54.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:54.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:33:54.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:54.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:33:54.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:33:54.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:33:54.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:33:54.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:33:54.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:33:54.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:33:54.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:33:54.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:54.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:33:54.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:33:54.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:33:54.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:33:54.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:33:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:33:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:33:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:33:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:33:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:33:54.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:33:54.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:33:54.369 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.369 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.369 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.369 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:577 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:54.369 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.923702+0000", 2024-04-24T14:33:54.369 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.339389513, 2024-04-24T14:33:54.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012522472, 2024-04-24T14:33:54.369 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 577 2024-04-24T14:33:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923702+0000", 2024-04-24T14:33:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923702+0000", 2024-04-24T14:33:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923701+0000", 2024-04-24T14:33:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923708+0000", 2024-04-24T14:33:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.233e-06 2024-04-24T14:33:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923723+0000", 2024-04-24T14:33:54.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5099000000000001e-05 2024-04-24T14:33:54.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923733+0000", 2024-04-24T14:33:54.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:33:54.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.935366+0000", 2024-04-24T14:33:54.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011632105 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.935446+0000", 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012000000000006e-05 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.936224+0000", 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077877899999999995 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.373 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.374 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:54.375 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:54.375 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:54.379+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:54.476 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:54.478+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:54.489 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:54.489 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:54.489 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:54.489 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:54.500 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:54.576 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:54.578+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:54.593 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:54.593 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:54.602 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:54.676 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:54.678+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:54.707 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records in 2024-04-24T14:33:54.707 INFO:tasks.workunit.client.0.smithi173.stderr:64+0 records out 2024-04-24T14:33:54.707 INFO:tasks.workunit.client.0.smithi173.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.753428 s, 89.1 MB/s 2024-04-24T14:33:54.708 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:54.709 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:54.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:54.709 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:54.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.709 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:379 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.709 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:54.709 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.049175922, 2024-04-24T14:33:54.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068078410000000002, 2024-04-24T14:33:54.709 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.709 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.709 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.710 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.710 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.710 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 379 2024-04-24T14:33:54.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.710 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:54.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:54.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:54.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:54.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664240+0000", 2024-04-24T14:33:54.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6539999999999999e-06 2024-04-24T14:33:54.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664245+0000", 2024-04-24T14:33:54.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2179999999999998e-06 2024-04-24T14:33:54.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664249+0000", 2024-04-24T14:33:54.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2279999999999998e-06 2024-04-24T14:33:54.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.713 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670245+0000", 2024-04-24T14:33:54.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059962430000000001 2024-04-24T14:33:54.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.713 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670400+0000", 2024-04-24T14:33:54.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015535399999999999 2024-04-24T14:33:54.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.714 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.714 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671046+0000", 2024-04-24T14:33:54.714 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064591599999999996 2024-04-24T14:33:54.714 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.714 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.714 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.714 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.714 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.714 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:380 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.714 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:33:54.714 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.049129295, 2024-04-24T14:33:54.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076160830000000001, 2024-04-24T14:33:54.715 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.715 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.715 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.715 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.715 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.715 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 380 2024-04-24T14:33:54.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.715 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:33:54.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:33:54.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664284+0000", 2024-04-24T14:33:54.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:54.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664287+0000", 2024-04-24T14:33:54.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9739999999999998e-06 2024-04-24T14:33:54.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664302+0000", 2024-04-24T14:33:54.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5214999999999999e-05 2024-04-24T14:33:54.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.718 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664307+0000", 2024-04-24T14:33:54.718 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6859999999999997e-06 2024-04-24T14:33:54.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.718 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671098+0000", 2024-04-24T14:33:54.718 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067912479999999997 2024-04-24T14:33:54.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.718 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.719 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.719 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671253+0000", 2024-04-24T14:33:54.719 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015427400000000001 2024-04-24T14:33:54.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.719 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.719 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.719 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671901+0000", 2024-04-24T14:33:54.719 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064845600000000005 2024-04-24T14:33:54.719 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.719 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.719 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.720 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.720 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.720 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.720 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:54.720 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.04909975, 2024-04-24T14:33:54.720 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:33:54.720 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.720 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.720 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.720 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.720 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.720 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:33:54.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.721 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.721 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.721 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:54.721 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.721 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.721 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:54.721 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.722 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.722 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:54.722 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:54.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.722 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.722 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:33:54.722 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:33:54.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:33:54.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:33:54.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:33:54.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:33:54.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:33:54.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:33:54.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.724 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.724 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.724 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:33:54.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:33:54.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.724 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.724 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.724 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:33:54.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:33:54.724 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.725 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.725 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.725 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.725 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.725 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:382 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.725 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:33:54.725 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.046098267, 2024-04-24T14:33:54.725 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067029910000000002, 2024-04-24T14:33:54.725 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.725 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.725 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.726 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.726 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.726 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 382 2024-04-24T14:33:54.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.726 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.726 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:33:54.726 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.727 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.727 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:33:54.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.727 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.727 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.727 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667315+0000", 2024-04-24T14:33:54.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:54.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.727 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.728 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.728 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667321+0000", 2024-04-24T14:33:54.728 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2099999999999998e-06 2024-04-24T14:33:54.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.728 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.728 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.728 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667336+0000", 2024-04-24T14:33:54.728 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4612e-05 2024-04-24T14:33:54.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.728 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.728 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.728 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667343+0000", 2024-04-24T14:33:54.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0119999999999999e-06 2024-04-24T14:33:54.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.729 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.729 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.729 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672957+0000", 2024-04-24T14:33:54.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056139700000000002 2024-04-24T14:33:54.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.729 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.729 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.729 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.673127+0000", 2024-04-24T14:33:54.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017044599999999999 2024-04-24T14:33:54.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.730 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.730 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.730 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674019+0000", 2024-04-24T14:33:54.730 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00089165499999999999 2024-04-24T14:33:54.730 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.730 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.730 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.730 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.730 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.731 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:54.731 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.046065223999999, 2024-04-24T14:33:54.731 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:33:54.731 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.731 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.731 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.731 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.731 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.731 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:33:54.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.731 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.731 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:54.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:54.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:54.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:54.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.733 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.733 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:33:54.733 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:33:54.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.733 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.733 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:33:54.733 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:33:54.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.734 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:33:54.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:33:54.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.734 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:33:54.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:33:54.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.734 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:33:54.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:33:54.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.735 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.735 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:33:54.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:33:54.735 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.735 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.735 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.736 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.736 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:54.736 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.046041038, 2024-04-24T14:33:54.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:33:54.736 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.736 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.736 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.736 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.736 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.736 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:33:54.736 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.737 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.737 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.737 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:54.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.737 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.737 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:54.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:54.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:54.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:33:54.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:33:54.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.739 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.739 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:33:54.739 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:33:54.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.739 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.739 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:33:54.739 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:33:54.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.739 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.739 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:33:54.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:33:54.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.740 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.740 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.740 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:33:54.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:33:54.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.740 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.740 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.740 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:33:54.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:33:54.740 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.741 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.741 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.741 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.741 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:33:54.741 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.046023654000001, 2024-04-24T14:33:54.741 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:33:54.741 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.741 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.741 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.741 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.742 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.742 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:33:54.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.742 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.742 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:33:54.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:33:54.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:33:54.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:54.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:33:54.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:33:54.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.744 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.744 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:33:54.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:33:54.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.744 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.744 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:33:54.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:33:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:33:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:33:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:33:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:33:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:33:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:33:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:399 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:33:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.031613519, 2024-04-24T14:33:54.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071317810000000002, 2024-04-24T14:33:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 399 2024-04-24T14:33:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:33:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:33:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681800+0000", 2024-04-24T14:33:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:54.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681804+0000", 2024-04-24T14:33:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.715e-06 2024-04-24T14:33:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681816+0000", 2024-04-24T14:33:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1851000000000001e-05 2024-04-24T14:33:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.749 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681820+0000", 2024-04-24T14:33:54.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6450000000000004e-06 2024-04-24T14:33:54.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.750 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.750 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.750 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688059+0000", 2024-04-24T14:33:54.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062390090000000002 2024-04-24T14:33:54.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.750 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.750 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.750 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688196+0000", 2024-04-24T14:33:54.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013721999999999999 2024-04-24T14:33:54.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688933+0000", 2024-04-24T14:33:54.751 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073605000000000003 2024-04-24T14:33:54.751 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.751 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.751 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.751 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.751 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:33:54.752 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.031587562, 2024-04-24T14:33:54.752 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:33:54.752 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.752 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.752 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.752 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.752 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.752 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:33:54.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.752 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.752 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.753 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.753 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:33:54.753 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.753 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.753 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.753 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.753 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:33:54.753 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.753 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.753 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.754 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.754 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:33:54.754 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:54.754 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.754 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.754 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.754 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:33:54.754 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:33:54.754 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.754 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.754 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:33:54.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:33:54.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:33:54.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:33:54.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:33:54.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:33:54.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.756 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.756 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.756 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:33:54.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:33:54.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.756 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.756 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.756 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:33:54.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:33:54.756 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.756 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.757 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.757 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.757 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:33:54.757 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.031555263, 2024-04-24T14:33:54.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:33:54.757 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.757 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.757 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.758 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.758 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.758 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:33:54.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.758 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.758 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.758 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.758 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:33:54.758 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.758 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.759 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.759 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:33:54.759 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.759 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.759 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.759 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.759 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:33:54.759 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:54.759 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.759 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.759 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.760 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:33:54.760 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:33:54.760 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.760 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.760 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.760 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:33:54.760 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:33:54.760 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.760 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.760 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.760 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:33:54.760 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:33:54.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.761 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.761 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.761 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:33:54.761 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:33:54.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.761 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.761 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.761 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:33:54.761 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:33:54.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.761 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.762 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.762 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:33:54.762 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:33:54.762 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.762 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.762 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.762 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.762 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.762 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:414 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.762 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:33:54.762 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.017359853, 2024-04-24T14:33:54.763 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069877910000000001, 2024-04-24T14:33:54.763 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.763 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.763 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.763 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.763 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.763 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 414 2024-04-24T14:33:54.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.763 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.763 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.763 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.763 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:33:54.764 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.764 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.764 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.764 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:33:54.764 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.764 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.764 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.764 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696053+0000", 2024-04-24T14:33:54.764 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:54.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.765 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.765 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.765 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696058+0000", 2024-04-24T14:33:54.765 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7509999999999999e-06 2024-04-24T14:33:54.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.765 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.765 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.765 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696074+0000", 2024-04-24T14:33:54.765 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5743000000000002e-05 2024-04-24T14:33:54.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.765 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.766 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.766 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696093+0000", 2024-04-24T14:33:54.766 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9508e-05 2024-04-24T14:33:54.766 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.766 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.766 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.766 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702314+0000", 2024-04-24T14:33:54.766 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062207010000000004 2024-04-24T14:33:54.766 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.766 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.767 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.767 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702431+0000", 2024-04-24T14:33:54.767 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011723299999999999 2024-04-24T14:33:54.767 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.767 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.767 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.767 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703042+0000", 2024-04-24T14:33:54.767 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000610752 2024-04-24T14:33:54.767 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.767 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.767 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.768 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.768 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.768 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.768 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:33:54.768 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.017302986000001, 2024-04-24T14:33:54.768 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:33:54.768 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.768 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.768 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.768 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.768 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.768 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:33:54.768 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.769 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.769 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.769 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.769 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:33:54.769 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.769 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.769 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.769 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.769 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:33:54.769 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.769 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.770 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.770 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.770 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:33:54.770 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:54.770 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.770 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.770 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.770 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:33:54.770 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:33:54.770 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.770 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.771 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.771 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:33:54.771 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:33:54.771 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.771 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.771 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.771 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:33:54.771 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:33:54.771 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.771 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.771 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.771 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:33:54.772 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:33:54.772 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.772 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.772 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.772 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:33:54.772 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:33:54.772 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.772 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.772 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.772 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:33:54.772 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:33:54.773 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.773 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.773 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.773 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.773 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.773 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.773 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:54.773 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.017273886, 2024-04-24T14:33:54.773 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:33:54.773 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.773 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.774 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.774 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.774 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.774 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:33:54.774 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.774 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.774 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.774 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.774 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:54.774 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.774 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.774 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.774 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.775 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:54.775 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.775 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.775 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.775 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.775 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:54.775 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:54.775 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.775 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.775 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.775 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:33:54.775 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:33:54.775 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.776 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.776 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.776 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:33:54.776 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:33:54.776 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.776 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.776 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.776 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:33:54.776 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:33:54.776 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.776 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.777 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.777 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:33:54.777 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:33:54.777 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.777 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.777 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.777 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:33:54.777 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:33:54.777 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.777 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.777 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.778 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:33:54.778 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:33:54.778 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.778 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.778 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.778 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.778 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.778 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.778 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:54.778 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.017247956, 2024-04-24T14:33:54.778 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:33:54.779 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.779 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.779 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.779 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.779 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.779 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:33:54.779 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.779 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.779 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.779 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.779 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:54.779 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.780 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.780 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.780 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.780 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:54.780 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.780 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.780 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.780 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.780 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:54.780 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:54.780 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.781 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.781 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.781 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:33:54.781 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:33:54.781 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.781 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.781 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.781 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:33:54.781 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:33:54.781 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.781 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.781 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.782 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:33:54.782 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:33:54.782 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.782 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.782 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.782 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:33:54.782 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:33:54.782 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.782 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.782 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.782 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:33:54.782 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:33:54.783 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.783 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.783 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.783 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:33:54.783 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:33:54.783 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.783 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.783 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.783 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.783 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.783 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:429 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3538944~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.783 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:33:54.784 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.002966516000001, 2024-04-24T14:33:54.784 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071866789999999996, 2024-04-24T14:33:54.784 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.784 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.784 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.784 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.784 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.784 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 429 2024-04-24T14:33:54.784 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.784 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.784 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.784 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.785 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:33:54.785 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.785 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.785 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.785 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.785 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:33:54.785 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.785 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.785 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.785 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.785 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710447+0000", 2024-04-24T14:33:54.785 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:54.786 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.786 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.786 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.786 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710451+0000", 2024-04-24T14:33:54.786 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5930000000000002e-06 2024-04-24T14:33:54.786 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.786 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.786 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.786 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710465+0000", 2024-04-24T14:33:54.786 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.411e-05 2024-04-24T14:33:54.786 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.787 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.787 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.787 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710470+0000", 2024-04-24T14:33:54.787 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3129999999999998e-06 2024-04-24T14:33:54.787 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.787 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.787 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.787 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716696+0000", 2024-04-24T14:33:54.787 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006225983 2024-04-24T14:33:54.787 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.787 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.787 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.788 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716901+0000", 2024-04-24T14:33:54.788 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020550200000000001 2024-04-24T14:33:54.788 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.788 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.788 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.788 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717634+0000", 2024-04-24T14:33:54.788 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073292099999999998 2024-04-24T14:33:54.788 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.788 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.788 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.788 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.788 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.789 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:430 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.789 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:33:54.789 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.001602208, 2024-04-24T14:33:54.789 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068326970000000004, 2024-04-24T14:33:54.789 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.789 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.789 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.789 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.789 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.789 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 430 2024-04-24T14:33:54.789 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.789 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.789 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.790 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.790 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:33:54.790 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.790 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.790 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.790 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.790 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:33:54.790 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.790 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.790 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.790 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.791 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711810+0000", 2024-04-24T14:33:54.791 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:54.791 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.791 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.791 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.791 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711822+0000", 2024-04-24T14:33:54.791 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2166e-05 2024-04-24T14:33:54.791 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.791 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.791 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.792 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711856+0000", 2024-04-24T14:33:54.792 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3816000000000003e-05 2024-04-24T14:33:54.792 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.792 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.792 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.792 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711873+0000", 2024-04-24T14:33:54.792 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6742999999999999e-05 2024-04-24T14:33:54.792 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.792 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.792 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.792 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717693+0000", 2024-04-24T14:33:54.793 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058204579999999997 2024-04-24T14:33:54.793 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.793 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.793 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.793 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717864+0000", 2024-04-24T14:33:54.793 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017128299999999999 2024-04-24T14:33:54.793 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.793 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.793 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.793 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718645+0000", 2024-04-24T14:33:54.793 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078033699999999998 2024-04-24T14:33:54.794 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.794 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.794 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.794 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.794 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.794 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.794 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:33:54.794 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.001527491999999, 2024-04-24T14:33:54.794 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:33:54.794 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.794 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:54.795 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.795 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.795 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.795 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:33:54.795 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.795 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.795 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.795 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.795 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:33:54.795 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.795 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.795 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.796 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.796 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:33:54.796 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.796 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.796 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.796 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:33:54.796 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:54.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.796 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.796 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.797 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:33:54.797 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:33:54.797 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.797 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.797 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.797 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:33:54.797 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:33:54.797 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.797 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.797 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.797 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:33:54.798 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:33:54.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.798 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.798 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.798 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:33:54.798 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:33:54.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.798 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.798 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.798 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:33:54.798 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:33:54.799 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.799 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.799 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.799 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:33:54.799 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:33:54.799 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.799 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.799 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.799 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.799 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.799 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.00149556, 2024-04-24T14:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.800 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.801 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:33:54.802 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:33:54.803 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.803 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.803 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.803 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:33:54.803 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:33:54.803 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.803 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.803 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.803 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:33:54.803 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.804 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.001428044000001, 2024-04-24T14:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.805 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.806 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.806 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:33:54.806 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.806 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.806 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.806 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.806 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:33:54.806 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.806 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.806 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.807 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:33:54.808 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:33:54.808 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.808 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.808 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.808 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:33:54.808 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:33:54.808 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.808 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.808 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.808 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:33:54.808 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:33:54.808 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.809 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.809 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.809 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:33:54.809 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:33:54.809 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.809 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.809 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.809 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.809 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.809 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:33:54.809 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:54.809 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.944015160999999, 2024-04-24T14:33:54.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:33:54.810 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:54.810 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:54.810 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:54.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:54.810 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:54.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:54.811 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:54.811 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:54.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.811 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:54.811 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:54.811 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:54.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.811 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:54.811 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.813 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:54.813 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:33:54.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:33:54.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.813 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:54.813 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:33:54.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:33:54.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.813 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:54.814 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:33:54.814 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:33:54.814 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:54.814 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:54.814 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:54.814 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:33:54.814 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:33:54.814 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.814 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.814 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.814 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:54.815 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:54.815 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:54.815 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [5, 4] dead_osds: [1] live_osds: [0, 2, 3, 4, 5] 2024-04-24T14:33:54.815 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:33:54.815 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-04-24T14:33:54.816 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-24T14:33:54.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-24T14:33:54.817 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false timeout 10 rbd bench testrbdnbd16809 --io-type write --io-size=1024 --io-total=1024 2024-04-24T14:33:54.818 INFO:tasks.workunit.client.0.smithi173.stderr:+ timeout 10 rbd bench testrbdnbd16809 --io-type write --io-size=1024 --io-total=1024 2024-04-24T14:33:54.818 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:54.819 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:54.818+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:54.820 INFO:tasks.ceph.osd.1:Started 2024-04-24T14:33:54.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:33:54.917 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:54.918+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:54.922 INFO:teuthology.orchestra.run.smithi090.stderr:no valid command found; 10 closest matches: 2024-04-24T14:33:54.922 INFO:teuthology.orchestra.run.smithi090.stderr:0 2024-04-24T14:33:54.922 INFO:teuthology.orchestra.run.smithi090.stderr:1 2024-04-24T14:33:54.922 INFO:teuthology.orchestra.run.smithi090.stderr:2 2024-04-24T14:33:54.923 INFO:teuthology.orchestra.run.smithi090.stderr:abort 2024-04-24T14:33:54.923 INFO:teuthology.orchestra.run.smithi090.stderr:assert 2024-04-24T14:33:54.923 INFO:teuthology.orchestra.run.smithi090.stderr:config diff 2024-04-24T14:33:54.923 INFO:teuthology.orchestra.run.smithi090.stderr:config diff get 2024-04-24T14:33:54.923 INFO:teuthology.orchestra.run.smithi090.stderr:config get 2024-04-24T14:33:54.923 INFO:teuthology.orchestra.run.smithi090.stderr:config help [] 2024-04-24T14:33:54.923 INFO:teuthology.orchestra.run.smithi090.stderr:config set ... 2024-04-24T14:33:54.923 INFO:teuthology.orchestra.run.smithi090.stderr:admin_socket: invalid command 2024-04-24T14:33:54.924 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-24T14:33:54.925 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-24T14:33:54.927 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:54.927 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:54.927 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:54.927 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:54.937 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:54.946 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:33:54.950+0000 7f97776be700 0 --2- 172.21.15.173:0/2545714807 >> v2:172.21.15.103:3300/0 conn(0x557c5959c140 0x557c59634e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-24T14:33:54.987 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:33:54.990+0000 7f97766bc700 -1 librbd::exclusive_lock::ImageDispatch: 0x7f97500020d0 handle_acquire_lock: failed to acquire exclusive lock: (30) Read-only file system 2024-04-24T14:33:54.987 INFO:tasks.workunit.client.0.smithi173.stderr:rbd: failed to flush: 2024-04-24T14:33:54.990+0000 7f97766bc700 -1 librbd::io::AioCompletion: 0x557c59699730 fail: (30) Read-only file system 2024-04-24T14:33:54.988 INFO:tasks.workunit.client.0.smithi173.stderr:(30) Read-only file system 2024-04-24T14:33:54.988 INFO:tasks.workunit.client.0.smithi173.stderr:bench failed: (30) Read-only file system 2024-04-24T14:33:54.997 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:54.997 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device /dev/nbd0 17833 2024-04-24T14:33:54.997 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=/dev/nbd0 2024-04-24T14:33:54.997 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=17833 2024-04-24T14:33:54.997 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:33:54.998 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:54.998 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:54.998 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:54.999 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:33:55.002 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:33:55.002 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:55.002 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:33:55.017 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:55.018+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:55.034 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:55.035 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:55.044 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:55.098 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:33:55.098 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^17833\b' 2024-04-24T14:33:55.098 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^17833\b' 2024-04-24T14:33:55.118 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:55.123+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:1 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.941932541, 2024-04-24T14:33:55.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002159392, 2024-04-24T14:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 1 2024-04-24T14:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201710+0000", 2024-04-24T14:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201731+0000", 2024-04-24T14:33:55.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0835999999999998e-05 2024-04-24T14:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201769+0000", 2024-04-24T14:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7472999999999998e-05 2024-04-24T14:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201796+0000", 2024-04-24T14:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6798999999999999e-05 2024-04-24T14:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201923+0000", 2024-04-24T14:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012770699999999999 2024-04-24T14:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203873+0000", 2024-04-24T14:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00194992 2024-04-24T14:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.144 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 15.071714558, 2024-04-24T14:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:33:55.148 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:52 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'51, mlcod=57'49)", 2024-04-24T14:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.061270+0000", 2024-04-24T14:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.0823762699999999, 2024-04-24T14:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024587599999999999, 2024-04-24T14:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061270+0000", 2024-04-24T14:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061270+0000", 2024-04-24T14:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061268+0000", 2024-04-24T14:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061425+0000", 2024-04-24T14:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015709400000000001 2024-04-24T14:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061469+0000", 2024-04-24T14:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4759000000000003e-05 2024-04-24T14:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061488+0000", 2024-04-24T14:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.821e-05 2024-04-24T14:33:55.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061634+0000", 2024-04-24T14:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014643300000000001 2024-04-24T14:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061712+0000", 2024-04-24T14:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8019e-05 2024-04-24T14:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.063621+0000", 2024-04-24T14:33:55.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001909004 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.063729+0000", 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107685 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.076899979, 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.153 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:33:55.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:33:55.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:33:55.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:33:55.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:33:55.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:33:55.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:33:55.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:33:55.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:33:55.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:33:55.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:33:55.157 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.157 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.157 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.158 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:33:55.158 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:33:55.158 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.065970721, 2024-04-24T14:33:55.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:33:55.158 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.158 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.158 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:33:55.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:33:55.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:33:55.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:55.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:33:55.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:33:55.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:33:55.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:33:55.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:33:55.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:33:55.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:33:55.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:33:55.162 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.162 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.162 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.162 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:33:55.162 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:33:55.162 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.05501464, 2024-04-24T14:33:55.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:33:55.162 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.162 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.163 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:33:55.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:33:55.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:33:55.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:55.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:33:55.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:33:55.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:33:55.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:33:55.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:33:55.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:33:55.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:33:55.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:33:55.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:33:55.166 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.166 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.166 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.167 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:33:55.167 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:33:55.167 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 1.0434452750000001, 2024-04-24T14:33:55.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:33:55.167 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.167 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.167 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:33:55.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:33:55.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:33:55.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:55.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:33:55.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:33:55.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:33:55.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:33:55.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:33:55.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:33:55.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:33:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:33:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:33:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:33:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:33:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:33:55.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:33:55.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:33:55.171 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.171 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.171 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.171 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:109 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'60, mlcod=62'58)", 2024-04-24T14:33:55.171 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:33:55.171 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.80033310999999996, 2024-04-24T14:33:55.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036290020000000001, 2024-04-24T14:33:55.172 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.172 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.172 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:33:55.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:33:55.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343311+0000", 2024-04-24T14:33:55.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:55.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343841+0000", 2024-04-24T14:33:55.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053003399999999997 2024-04-24T14:33:55.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343876+0000", 2024-04-24T14:33:55.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4479e-05 2024-04-24T14:33:55.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343888+0000", 2024-04-24T14:33:55.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1623e-05 2024-04-24T14:33:55.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343996+0000", 2024-04-24T14:33:55.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108028 2024-04-24T14:33:55.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.344044+0000", 2024-04-24T14:33:55.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8081999999999998e-05 2024-04-24T14:33:55.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.346872+0000", 2024-04-24T14:33:55.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028280789999999998 2024-04-24T14:33:55.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.346942+0000", 2024-04-24T14:33:55.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0405000000000001e-05 2024-04-24T14:33:55.175 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.175 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.175 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.176 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:33:55.176 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:33:55.176 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.79032561599999995, 2024-04-24T14:33:55.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:33:55.176 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.176 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.176 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:33:55.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:33:55.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:33:55.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:55.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:33:55.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:33:55.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:33:55.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:33:55.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:33:55.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:33:55.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.77958044800000004, 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.180 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:33:55.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:33:55.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:33:55.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:33:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:33:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:33:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:33:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:33:55.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:33:55.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.183 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:33:55.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:33:55.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.183 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:33:55.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:33:55.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:33:55.184 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:33:55.185 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.76721417199999997, 2024-04-24T14:33:55.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:33:55.185 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.185 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.185 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:33:55.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:33:55.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:33:55.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:55.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:33:55.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:33:55.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:33:55.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:33:55.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:33:55.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:33:55.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:33:55.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:33:55.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:33:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:33:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:33:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:33:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:33:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:33:55.189 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.189 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.189 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.189 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:33:55.189 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:33:55.189 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.64227805100000002, 2024-04-24T14:33:55.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:33:55.189 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.189 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.189 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:33:55.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.190 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:33:55.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.190 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:33:55.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:55.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:33:55.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:33:55.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:33:55.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:33:55.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:33:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:33:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:33:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:33:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:33:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:33:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:33:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:33:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:33:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:33:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:138 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'70, mlcod=62'68)", 2024-04-24T14:33:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:33:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.63189189700000004, 2024-04-24T14:33:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036613330000000001, 2024-04-24T14:33:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:33:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:33:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511753+0000", 2024-04-24T14:33:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512370+0000", 2024-04-24T14:33:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061776500000000005 2024-04-24T14:33:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512395+0000", 2024-04-24T14:33:55.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4549000000000002e-05 2024-04-24T14:33:55.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512404+0000", 2024-04-24T14:33:55.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1130000000000007e-06 2024-04-24T14:33:55.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512517+0000", 2024-04-24T14:33:55.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112679 2024-04-24T14:33:55.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512581+0000", 2024-04-24T14:33:55.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4274000000000002e-05 2024-04-24T14:33:55.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.515303+0000", 2024-04-24T14:33:55.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027220640000000002 2024-04-24T14:33:55.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.515416+0000", 2024-04-24T14:33:55.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112644 2024-04-24T14:33:55.197 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:33:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:33:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.62289020799999995, 2024-04-24T14:33:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:33:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:33:55.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:33:55.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:33:55.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:55.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:33:55.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:33:55.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.200 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:33:55.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:33:55.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.200 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:33:55.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:33:55.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:33:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:33:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:33:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:33:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:33:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.61316236499999999, 2024-04-24T14:33:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:33:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.203 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.203 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:33:55.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:33:55.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:33:55.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:55.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:33:55.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:33:55.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:33:55.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:33:55.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:33:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:33:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:33:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:33:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:33:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:33:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:33:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:33:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:33:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:33:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:171 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'75, mlcod=62'73)", 2024-04-24T14:33:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.675580+0000", 2024-04-24T14:33:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.46806652399999998, 2024-04-24T14:33:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002272273, 2024-04-24T14:33:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675580+0000", 2024-04-24T14:33:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675580+0000", 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675578+0000", 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675717+0000", 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013899999999999999 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675758+0000", 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1696999999999997e-05 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675775+0000", 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6430999999999999e-05 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675878+0000", 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000102749 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675940+0000", 2024-04-24T14:33:55.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2794999999999998e-05 2024-04-24T14:33:55.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.210 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.677755+0000", 2024-04-24T14:33:55.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0018146060000000001 2024-04-24T14:33:55.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.210 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.677852+0000", 2024-04-24T14:33:55.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7054999999999998e-05 2024-04-24T14:33:55.210 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.210 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.211 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.211 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:172 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'76, mlcod=62'74)", 2024-04-24T14:33:55.211 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:33:55.211 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.46289501100000002, 2024-04-24T14:33:55.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032080400000000001, 2024-04-24T14:33:55.211 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.211 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.211 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:33:55.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:33:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680750+0000", 2024-04-24T14:33:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681277+0000", 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052776400000000001 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681304+0000", 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6948e-05 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681315+0000", 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0648e-05 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681413+0000", 2024-04-24T14:33:55.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7926999999999999e-05 2024-04-24T14:33:55.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.214 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681470+0000", 2024-04-24T14:33:55.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7401999999999997e-05 2024-04-24T14:33:55.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.214 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.683923+0000", 2024-04-24T14:33:55.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024526790000000001 2024-04-24T14:33:55.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.683959+0000", 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6327e-05 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.45305268500000001, 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:33:55.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:33:55.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:33:55.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:55.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:33:55.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:33:55.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:33:55.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:33:55.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:33:55.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:33:55.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:33:55.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:33:55.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:33:55.219 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.219 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.219 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.219 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:33:55.219 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:33:55.219 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.44332368900000002, 2024-04-24T14:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:33:55.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.223 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:178 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'82, mlcod=62'80)", 2024-04-24T14:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.43285631299999999, 2024-04-24T14:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029527640000000001, 2024-04-24T14:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710789+0000", 2024-04-24T14:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711538+0000", 2024-04-24T14:33:55.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00074893099999999999 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711565+0000", 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7526999999999999e-05 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711576+0000", 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0553999999999999e-05 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711633+0000", 2024-04-24T14:33:55.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7258e-05 2024-04-24T14:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711653+0000", 2024-04-24T14:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0460999999999999e-05 2024-04-24T14:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.227 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.713712+0000", 2024-04-24T14:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020587790000000002 2024-04-24T14:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:55.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.713743+0000", 2024-04-24T14:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0519999999999999e-05 2024-04-24T14:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:55.230 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:55.232 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:55.232 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:33:55.232 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *17833\b' 2024-04-24T14:33:55.232 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *17833\b' 2024-04-24T14:33:55.232 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:55.232 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:33:55.233 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:55.233 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:55.235+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:55.282 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:33:55.332 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:55.335+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:55.341 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:55.341 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:55.341 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:55.341 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:55.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:55.412 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:55.430 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:55.433 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:55.434+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:55.439 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:33:55.443+0000 7f229e4073c0 -1 Falling back to public interface 2024-04-24T14:33:55.449 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:55.450 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:55.451 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:55.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:55.475 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:55.496 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:55.496 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:55.497 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:55.499 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:55.533 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:55.535+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:55.550 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:55.550 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:55.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:55.634 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:55.639+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:55.665 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:33:55.665 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd bench testrbdnbd16809 --io-type write --io-size=1024 --io-total=1024 2024-04-24T14:33:55.708 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:55.708 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:55.717 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:55.735 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:55.738+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:55.738 INFO:tasks.workunit.client.0.smithi173.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-04-24T14:33:55.741 INFO:tasks.workunit.client.0.smithi173.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-04-24T14:33:55.741 INFO:tasks.workunit.client.0.smithi173.stdout:elapsed: 0 ops: 1 ops/sec: inf bytes/sec: 0 B/s 2024-04-24T14:33:55.754 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:33:55.754 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:33:55.755 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:33:55.757 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:33:55.757 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:33:55.760 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:33:55.760 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:33:55.760 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:33:55.813 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:55.813 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:55.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:55.836 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:55.839+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:55.908 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:33:55.908 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:33:55.909 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:33:55.909 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:33:55.910 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:33:55.910 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:33:55.918 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:55.918 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:55.927 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:55.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T14:33:55.971 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:55.974+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:55.981 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=17946 2024-04-24T14:33:55.981 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 17946 2024-04-24T14:33:55.981 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 17946 -C rbd-nbd 2024-04-24T14:33:55.992 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:33:55.992 INFO:tasks.workunit.client.0.smithi173.stdout: 17946 ? 00:00:00 rbd-nbd 2024-04-24T14:33:55.993 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device testrbdnbd16809 17946 2024-04-24T14:33:55.993 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=testrbdnbd16809 2024-04-24T14:33:55.993 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=17946 2024-04-24T14:33:55.993 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd16809 2024-04-24T14:33:55.993 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:55.993 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:55.994 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:55.995 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:33:55.997 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:33:55.997 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:55.997 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd16809 2024-04-24T14:33:56.038 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:56.038+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:56.039 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:56.039 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:56.049 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:56.092 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:33:56.092 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^17946\b' 2024-04-24T14:33:56.092 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^17946\b' 2024-04-24T14:33:56.139 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:56.142+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:56.146 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:56.146 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:56.149 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:56.150 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:33:56.150 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *17946\b' 2024-04-24T14:33:56.150 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *17946\b' 2024-04-24T14:33:56.155 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:56.156 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:56.156 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:33:56.240 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:56.243+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:56.304 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:56.313 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:56.341 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:56.343+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:56.412 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:56.421 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:56.441 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:56.442+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:56.523 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:56.523 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:56.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:56.542 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:56.547+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:56.643 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:56.642+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:56.658 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:33:56.658 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd snap create rbd/testrbdnbd16809@snap 2024-04-24T14:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:56.680 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:56.689 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:56.743 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:56.747+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:56.788 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:56.797 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:56.824 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-24T14:33:56.833 INFO:tasks.workunit.client.0.smithi173.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-04-24T14:33:56.844 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:56.846+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:56.848 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd16809@snap 2024-04-24T14:33:56.848 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:33:56.849 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:33:56.850 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:33:56.851 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:33:56.853 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:33:56.853 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:33:56.854 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd16809@snap 2024-04-24T14:33:56.897 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:56.897 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:56.906 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:56.944 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:56.946+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:57.008 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:33:57.008 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:33:57.008 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:33:57.008 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:33:57.009 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:57.009 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:57.010 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:33:57.010 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:33:57.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:57.059 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:57.062+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:57.082 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=18045 2024-04-24T14:33:57.082 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 18045 2024-04-24T14:33:57.082 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 18045 -C rbd-nbd 2024-04-24T14:33:57.090 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:33:57.090 INFO:tasks.workunit.client.0.smithi173.stdout: 18045 ? 00:00:00 rbd-nbd 2024-04-24T14:33:57.091 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device testrbdnbd16809@snap 18045 2024-04-24T14:33:57.091 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=testrbdnbd16809@snap 2024-04-24T14:33:57.091 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=18045 2024-04-24T14:33:57.091 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd16809@snap 2024-04-24T14:33:57.091 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:57.091 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:57.092 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:57.093 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:33:57.095 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:33:57.095 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:57.095 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd16809@snap 2024-04-24T14:33:57.124 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:57.125 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:57.134 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:57.146 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:57.146+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:57.185 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:33:57.185 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^18045\b' 2024-04-24T14:33:57.185 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^18045\b' 2024-04-24T14:33:57.230 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:57.230 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:57.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:57.239 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:57.240 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:33:57.240 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *18045\b' 2024-04-24T14:33:57.240 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *18045\b' 2024-04-24T14:33:57.247 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:57.250+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:57.254 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:57.254 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:33:57.335 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:57.335 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:57.344 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:57.347 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:57.351+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:57.446 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:57.446 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:57.447 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:57.451+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:57.455 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:57.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:33:57.549 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:57.554+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:57.557 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:33:57.557 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:33:57.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:57.627 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:33:57.644 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:33:57.649 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:57.651+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:57.665 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:33:57.667 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:192 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.200155970000001, 2024-04-24T14:33:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010036329, 2024-04-24T14:33:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471099+0000", 2024-04-24T14:33:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471103+0000", 2024-04-24T14:33:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8800000000000001e-06 2024-04-24T14:33:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471111+0000", 2024-04-24T14:33:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2930000000000001e-06 2024-04-24T14:33:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471116+0000", 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1800000000000004e-06 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480338+0000", 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092213069999999998 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480467+0000", 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012901500000000001 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481136+0000", 2024-04-24T14:33:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066948400000000003 2024-04-24T14:33:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.200132933999999, 2024-04-24T14:33:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:33:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:33:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:33:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:33:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:33:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:33:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:33:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:33:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:33:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.824776231, 2024-04-24T14:33:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:33:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:33:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:33:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:33:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:33:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:33:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:33:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:33:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:33:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:33:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:526 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.823388708, 2024-04-24T14:33:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010568783999999999, 2024-04-24T14:33:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 526 2024-04-24T14:33:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847866+0000", 2024-04-24T14:33:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847871+0000", 2024-04-24T14:33:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1590000000000001e-06 2024-04-24T14:33:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847883+0000", 2024-04-24T14:33:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.172e-05 2024-04-24T14:33:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847890+0000", 2024-04-24T14:33:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6950000000000003e-06 2024-04-24T14:33:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857418+0000", 2024-04-24T14:33:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095285909999999995 2024-04-24T14:33:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857588+0000", 2024-04-24T14:33:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169541 2024-04-24T14:33:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858436+0000", 2024-04-24T14:33:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084814700000000005 2024-04-24T14:33:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.823348077, 2024-04-24T14:33:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:33:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:33:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:33:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:33:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:33:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:33:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:33:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:33:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:33:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:33:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:33:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:33:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:33:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:33:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:33:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.823320988000001, 2024-04-24T14:33:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:33:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:33:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:33:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:33:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:33:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:33:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:33:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:33:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:33:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:33:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:33:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:33:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:33:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:33:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:33:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.823297021, 2024-04-24T14:33:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:33:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:33:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:33:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:33:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:33:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:33:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:33:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:33:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:33:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:33:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:33:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:33:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:33:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:33:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:33:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.809313041999999, 2024-04-24T14:33:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:33:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:33:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:33:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:33:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:33:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:33:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:33:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:33:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:33:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:33:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:33:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:33:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:33:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:33:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:33:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:33:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:33:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:33:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:33:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:33:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:33:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:33:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:33:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.809196731, 2024-04-24T14:33:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:33:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:33:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:33:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:33:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:33:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:33:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:33:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:33:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:33:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:33:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:33:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:33:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:33:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:33:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:33:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:33:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:33:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:33:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.809149073, 2024-04-24T14:33:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:33:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:33:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:33:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:33:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:33:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:33:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:33:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:33:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:33:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:33:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:33:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:33:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:33:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:33:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:33:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:33:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:33:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:33:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:33:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:33:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.805111775, 2024-04-24T14:33:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:33:57.721 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.721 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.721 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:33:57.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.721 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:57.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:57.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:33:57.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:57.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:33:57.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:33:57.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:33:57.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:33:57.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:33:57.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:33:57.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:33:57.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:33:57.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:57.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:33:57.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:33:57.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:33:57.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:33:57.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:57.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:33:57.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:33:57.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:33:57.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:33:57.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:33:57.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:33:57.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:33:57.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:33:57.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.726 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.726 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.726 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:57.727 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.805048016000001, 2024-04-24T14:33:57.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:33:57.727 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.727 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.727 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:33:57.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.727 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:33:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:33:57.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:33:57.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:33:57.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:33:57.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:33:57.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:33:57.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:33:57.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:33:57.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:57.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:33:57.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:33:57.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:33:57.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:33:57.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:57.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:33:57.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:33:57.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:33:57.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:33:57.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:33:57.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:33:57.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:33:57.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:33:57.732 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.732 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.732 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.732 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.805022526, 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:57.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:57.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:57.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:57.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:33:57.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:33:57.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:33:57.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:33:57.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:33:57.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:33:57.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:33:57.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:33:57.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:57.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:33:57.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:33:57.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:33:57.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:33:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:33:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:33:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:33:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:33:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:33:57.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:33:57.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:33:57.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:33:57.738 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.738 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.738 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.738 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.804998776, 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:57.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:57.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:33:57.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:57.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:33:57.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:33:57.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:57.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:33:57.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:33:57.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:33:57.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:33:57.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:57.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:33:57.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:33:57.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:33:57.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:33:57.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:33:57.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.804960263, 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:33:57.744 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.745 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.745 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.745 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.745 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.745 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:33:57.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.745 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:57.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:57.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:33:57.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:57.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:33:57.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:33:57.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:33:57.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:33:57.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:33:57.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:33:57.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:33:57.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:33:57.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:57.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:33:57.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:33:57.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:33:57.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:33:57.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:57.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:33:57.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:33:57.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:33:57.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:33:57.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:33:57.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:33:57.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:33:57.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:33:57.750 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.750 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.750 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.750 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.750 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:33:57.750 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.804932065999999, 2024-04-24T14:33:57.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:33:57.751 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.751 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.751 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.751 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.751 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.751 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:33:57.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.751 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:33:57.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:33:57.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:33:57.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:33:57.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:33:57.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:33:57.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:33:57.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:57.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:33:57.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:33:57.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:33:57.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:33:57.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:57.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:33:57.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:33:57.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:33:57.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:33:57.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:33:57.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:33:57.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:33:57.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:33:57.756 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.756 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.756 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.756 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.756 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:33:57.756 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.804910609, 2024-04-24T14:33:57.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:33:57.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:33:57.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:57.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:33:57.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:33:57.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:33:57.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:33:57.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:33:57.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:33:57.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:33:57.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:33:57.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:57.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:33:57.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:33:57.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:33:57.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:33:57.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:33:57.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:33:57.762 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.762 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.762 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.762 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.762 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:33:57.762 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.804891767000001, 2024-04-24T14:33:57.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:33:57.763 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.763 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.763 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:33:57.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.763 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:33:57.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:33:57.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:33:57.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:57.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:33:57.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:33:57.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:33:57.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:33:57.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:33:57.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:33:57.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:33:57.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:33:57.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:57.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:33:57.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:33:57.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:33:57.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:33:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:33:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:33:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:33:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:33:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:561 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.762929146999999, 2024-04-24T14:33:57.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01066806, 2024-04-24T14:33:57.769 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.769 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:57.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:57.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:57.769 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 561 2024-04-24T14:33:57.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.769 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:33:57.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:33:57.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908326+0000", 2024-04-24T14:33:57.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:57.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908329+0000", 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.802e-06 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908338+0000", 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6260000000000001e-06 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908345+0000", 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1389999999999998e-06 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.917978+0000", 2024-04-24T14:33:57.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096329840000000007 2024-04-24T14:33:57.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918178+0000", 2024-04-24T14:33:57.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019999199999999999 2024-04-24T14:33:57.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918995+0000", 2024-04-24T14:33:57.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081708099999999999 2024-04-24T14:33:57.772 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.772 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.773 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.773 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:64 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:33:57.773 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:54.123327+0000", 2024-04-24T14:33:57.773 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.547928969, 2024-04-24T14:33:57.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099417059999999998, 2024-04-24T14:33:57.773 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:57.773 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:57.773 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:57.773 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:33:57.773 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:33:57.773 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 64 2024-04-24T14:33:57.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.774 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:57.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:57.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:54.123327+0000", 2024-04-24T14:33:57.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:57.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:54.123327+0000", 2024-04-24T14:33:57.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:57.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:57.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:54.123325+0000", 2024-04-24T14:33:57.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:57.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:57.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:54.129851+0000", 2024-04-24T14:33:57.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.006526263 2024-04-24T14:33:57.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:57.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:54.129873+0000", 2024-04-24T14:33:57.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1699999999999999e-05 2024-04-24T14:33:57.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:57.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:54.129883+0000", 2024-04-24T14:33:57.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0134e-05 2024-04-24T14:33:57.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:57.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:54.129962+0000", 2024-04-24T14:33:57.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8841e-05 2024-04-24T14:33:57.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:57.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:54.130120+0000", 2024-04-24T14:33:57.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000158142 2024-04-24T14:33:57.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:33:57.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:54.133075+0000", 2024-04-24T14:33:57.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029552200000000002 2024-04-24T14:33:57.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:57.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:54.133155+0000", 2024-04-24T14:33:57.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9334999999999999e-05 2024-04-24T14:33:57.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:57.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:57.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:57.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:54.133269+0000", 2024-04-24T14:33:57.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114023 2024-04-24T14:33:57.778 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.778 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.778 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.778 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:57.778 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:57.778 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:57.780 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:57.782+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:57.780 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:33:57.780 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd snap ls rbd/testrbdnbd16809 2024-04-24T14:33:57.780 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '$2 == "snap" {print $1}' 2024-04-24T14:33:57.781 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:33:57.782 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:33:57.782 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:33:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:33:57.783 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:33:57.783 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:33:57.783 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:33:57.783 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:33:57.783 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:33:57.783 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:33:57.784 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:33:57.824 INFO:tasks.workunit.client.0.smithi173.stderr:+ SNAPID=4 2024-04-24T14:33:57.824 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map --snap-id 4 rbd/testrbdnbd16809 2024-04-24T14:33:57.824 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:33:57.825 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:33:57.827 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:33:57.828 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:33:57.830 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:33:57.830 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:33:57.830 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 4 rbd/testrbdnbd16809 2024-04-24T14:33:57.881 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:57.882+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:57.894 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:57.894 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:57.894 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:57.894 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:57.903 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:33:57.981 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:57.983+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:57.985 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:33:57.985 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:33:57.985 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:33:57.985 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:33:57.986 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:33:57.986 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:33:58.001 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:58.001 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:58.001 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:58.001 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:58.001 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:58.010 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:33:58.057 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=18146 2024-04-24T14:33:58.057 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 18146 2024-04-24T14:33:58.057 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 18146 -C rbd-nbd 2024-04-24T14:33:58.070 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:33:58.070 INFO:tasks.workunit.client.0.smithi173.stdout: 18146 ? 00:00:00 rbd-nbd 2024-04-24T14:33:58.071 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device '--snap-id 4 testrbdnbd16809' 18146 2024-04-24T14:33:58.071 INFO:tasks.workunit.client.0.smithi173.stderr:+ local 'args=--snap-id 4 testrbdnbd16809' 2024-04-24T14:33:58.071 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=18146 2024-04-24T14:33:58.071 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 4 testrbdnbd16809 2024-04-24T14:33:58.071 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:58.072 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:58.073 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:58.073 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:33:58.075 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:33:58.075 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:58.075 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 4 testrbdnbd16809 2024-04-24T14:33:58.082 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:58.087+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:58.113 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:58.113 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:58.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:58.114 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:58.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.114 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.114 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:58.114 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.790284791999994, 2024-04-24T14:33:58.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.01227051, 2024-04-24T14:33:58.114 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.114 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.114 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.114 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.114 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.114 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 37 2024-04-24T14:33:58.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.115 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:58.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326063+0000", 2024-04-24T14:33:58.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326062+0000", 2024-04-24T14:33:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326068+0000", 2024-04-24T14:33:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7939999999999999e-06 2024-04-24T14:33:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326090+0000", 2024-04-24T14:33:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2368e-05 2024-04-24T14:33:58.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.326095+0000", 2024-04-24T14:33:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2830000000000001e-06 2024-04-24T14:33:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333599+0000", 2024-04-24T14:33:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007503567 2024-04-24T14:33:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333697+0000", 2024-04-24T14:33:58.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7850000000000007e-05 2024-04-24T14:33:58.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.333835+0000", 2024-04-24T14:33:58.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000138403 2024-04-24T14:33:58.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.334700+0000", 2024-04-24T14:33:58.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086484300000000001 2024-04-24T14:33:58.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338195+0000", 2024-04-24T14:33:58.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034952160000000002 2024-04-24T14:33:58.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338246+0000", 2024-04-24T14:33:58.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0605000000000001e-05 2024-04-24T14:33:58.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.338333+0000", 2024-04-24T14:33:58.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7274999999999994e-05 2024-04-24T14:33:58.120 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.120 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.120 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.121 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.121 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:58.121 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.490769850000007, 2024-04-24T14:33:58.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015437309, 2024-04-24T14:33:58.121 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.121 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.121 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.121 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.121 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.121 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 109 2024-04-24T14:33:58.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.121 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:58.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625578+0000", 2024-04-24T14:33:58.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625576+0000", 2024-04-24T14:33:58.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:58.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625717+0000", 2024-04-24T14:33:58.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000140846 2024-04-24T14:33:58.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625733+0000", 2024-04-24T14:33:58.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5923e-05 2024-04-24T14:33:58.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625741+0000", 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6000000000000001e-06 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.630991+0000", 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052505670000000003 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631256+0000", 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026518100000000002 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.631517+0000", 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00026015400000000002 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635089+0000", 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0035724039999999999 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640920+0000", 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058309130000000001 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640976+0000", 2024-04-24T14:33:58.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.5787e-05 2024-04-24T14:33:58.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641015+0000", 2024-04-24T14:33:58.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9169999999999999e-05 2024-04-24T14:33:58.126 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.126 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.126 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.126 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.126 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.490544082, 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:33:58.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:33:58.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:58.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:33:58.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:33:58.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:33:58.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:33:58.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:33:58.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:33:58.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:33:58.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:33:58.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:33:58.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:33:58.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:33:58.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:33:58.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:33:58.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.490366995000002, 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.132 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.133 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.133 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:33:58.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.133 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:58.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:33:58.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:33:58.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:58.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:33:58.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:33:58.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:33:58.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:33:58.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:33:58.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:33:58.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:33:58.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:33:58.137 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.137 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.138 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.138 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.138 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:58.138 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.490263080999995, 2024-04-24T14:33:58.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:33:58.138 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.138 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.138 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.138 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.138 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.138 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:33:58.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.139 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:58.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:33:58.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:33:58.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:58.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:33:58.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:33:58.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:33:58.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:33:58.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:33:58.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:33:58.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:33:58.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:33:58.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:33:58.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:33:58.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.143 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.489966761000005, 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015460161, 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 113 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626381+0000", 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626379+0000", 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626391+0000", 2024-04-24T14:33:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2043e-05 2024-04-24T14:33:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626413+0000", 2024-04-24T14:33:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2195999999999999e-05 2024-04-24T14:33:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626424+0000", 2024-04-24T14:33:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.044e-05 2024-04-24T14:33:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636076+0000", 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.009652094 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636149+0000", 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.3446000000000001e-05 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636259+0000", 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010984600000000001 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641194+0000", 2024-04-24T14:33:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049352900000000002 2024-04-24T14:33:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641705+0000", 2024-04-24T14:33:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051112599999999998 2024-04-24T14:33:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641793+0000", 2024-04-24T14:33:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7674000000000001e-05 2024-04-24T14:33:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641841+0000", 2024-04-24T14:33:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7799000000000001e-05 2024-04-24T14:33:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.489874630000003, 2024-04-24T14:33:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:33:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:33:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:33:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:33:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:33:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:33:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:33:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:33:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:33:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:33:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:33:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:33:58.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:33:58.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:33:58.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:33:58.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:33:58.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:33:58.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.489789418000001, 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.155 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:33:58.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:58.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:33:58.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:33:58.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:33:58.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:33:58.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:33:58.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:33:58.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:33:58.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:33:58.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:33:58.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:33:58.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:33:58.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:33:58.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:33:58.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:33:58.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:33:58.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:33:58.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:33:58.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:33:58.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:33:58.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:33:58.160 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.160 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.161 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.161 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.161 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:58.161 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.489552536999994, 2024-04-24T14:33:58.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:33:58.161 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.161 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.161 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.161 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.161 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.161 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:33:58.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.162 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:58.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:33:58.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:33:58.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:58.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:33:58.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:33:58.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:33:58.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:33:58.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:33:58.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:33:58.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:33:58.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:33:58.166 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.166 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.166 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.489471015000007, 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:58.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:33:58.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:33:58.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:33:58.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:33:58.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:33:58.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:33:58.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:33:58.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:33:58.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:33:58.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:33:58.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:33:58.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:33:58.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:33:58.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:33:58.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:33:58.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:33:58.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:33:58.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:33:58.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:33:58.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:33:58.172 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.172 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.172 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.172 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.172 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:58.172 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.057712604000002, 2024-04-24T14:33:58.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013508398, 2024-04-24T14:33:58.173 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.173 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.173 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.173 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.173 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.173 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 136 2024-04-24T14:33:58.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.173 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:58.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058635+0000", 2024-04-24T14:33:58.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058634+0000", 2024-04-24T14:33:58.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:58.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058641+0000", 2024-04-24T14:33:58.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7970000000000003e-06 2024-04-24T14:33:58.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058651+0000", 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7999999999999993e-06 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058656+0000", 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0810000000000001e-06 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066281+0000", 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076252730000000001 2024-04-24T14:33:58.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066462+0000", 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000181256 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.066670+0000", 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020719600000000001 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071021+0000", 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043515569999999998 2024-04-24T14:33:58.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072091+0000", 2024-04-24T14:33:58.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00107004 2024-04-24T14:33:58.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072120+0000", 2024-04-24T14:33:58.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8812000000000001e-05 2024-04-24T14:33:58.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072143+0000", 2024-04-24T14:33:58.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3149000000000001e-05 2024-04-24T14:33:58.178 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.178 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.178 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.178 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.178 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:58.178 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.057671100999997, 2024-04-24T14:33:58.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015008984, 2024-04-24T14:33:58.178 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.178 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.178 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.178 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.179 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.179 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 137 2024-04-24T14:33:58.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.179 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:58.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:58.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058676+0000", 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058765+0000", 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9747000000000007e-05 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058773+0000", 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.038e-06 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058779+0000", 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9529999999999996e-06 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067654+0000", 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088746129999999999 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067774+0000", 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119856 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.067978+0000", 2024-04-24T14:33:58.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000204305 2024-04-24T14:33:58.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072880+0000", 2024-04-24T14:33:58.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049020369999999997 2024-04-24T14:33:58.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073637+0000", 2024-04-24T14:33:58.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075680699999999997 2024-04-24T14:33:58.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.183 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073666+0000", 2024-04-24T14:33:58.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8581999999999998e-05 2024-04-24T14:33:58.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.183 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.073685+0000", 2024-04-24T14:33:58.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9675000000000001e-05 2024-04-24T14:33:58.183 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.183 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.184 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.184 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.184 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:58.184 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.057558667999999, 2024-04-24T14:33:58.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:33:58.184 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.184 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.184 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.184 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.184 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.184 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:33:58.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:33:58.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:58.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:33:58.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:33:58.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:33:58.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:33:58.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:33:58.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:33:58.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:33:58.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:33:58.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:33:58.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:33:58.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:33:58.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:33:58.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:33:58.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:33:58.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:33:58.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:33:58.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.189 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.057530368000002, 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:58.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:33:58.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:33:58.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:58.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:33:58.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:33:58.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:33:58.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:33:58.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:33:58.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:33:58.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:33:58.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:33:58.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:33:58.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:33:58.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:33:58.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:33:58.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:33:58.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:33:58.195 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.195 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.195 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.195 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.195 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:58.195 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.057442664000007, 2024-04-24T14:33:58.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:33:58.195 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.195 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.195 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:33:58.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:33:58.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:33:58.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:33:58.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:33:58.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:33:58.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.057319863000004, 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.201 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:58.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.202 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.202 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:33:58.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.202 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.202 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:33:58.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:58.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.202 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.206 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:33:58.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:33:58.206 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.206 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.206 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.206 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.206 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.206 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-04-24T14:33:58.206 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:58.206 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 94.414079931000003, 2024-04-24T14:33:58.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014907768, 2024-04-24T14:33:58.206 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.206 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.206 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 159 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702267+0000", 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702266+0000", 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702279+0000", 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2969999999999999e-05 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702299+0000", 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0259999999999999e-05 2024-04-24T14:33:58.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.702309+0000", 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0056e-05 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.710987+0000", 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0086779849999999992 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711095+0000", 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107817 2024-04-24T14:33:58.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.210 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:33:58.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.711326+0000", 2024-04-24T14:33:58.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000231478 2024-04-24T14:33:58.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.210 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:33:58.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.712510+0000", 2024-04-24T14:33:58.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001183906 2024-04-24T14:33:58.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.210 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:58.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717057+0000", 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045463389999999999 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717126+0000", 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8937999999999996e-05 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.717175+0000", 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9688999999999998e-05 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.212 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:33:58.212 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:58.212 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 25.751075303, 2024-04-24T14:33:58.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:33:58.212 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.212 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.212 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.212 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:33:58.212 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:33:58.212 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:33:58.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.212 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.214 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:33:58.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:33:58.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.214 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:33:58.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:33:58.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.214 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:33:58.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:33:58.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:33:58.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:33:58.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:33:58.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:33:58.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:33:58.216 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.216 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.216 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.056951303999998, 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:58.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:33:58.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:33:58.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:58.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:33:58.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:33:58.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:33:58.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:33:58.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:33:58.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:33:58.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:33:58.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:33:58.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:33:58.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:33:58.222 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.222 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.222 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.222 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:577 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:58.222 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.923702+0000", 2024-04-24T14:33:58.222 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.192645560999999, 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012522472, 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 577 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923702+0000", 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923702+0000", 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923701+0000", 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923708+0000", 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.233e-06 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923723+0000", 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5099000000000001e-05 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.923733+0000", 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.935366+0000", 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011632105 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.935446+0000", 2024-04-24T14:33:58.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012000000000006e-05 2024-04-24T14:33:58.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.936224+0000", 2024-04-24T14:33:58.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077877899999999995 2024-04-24T14:33:58.226 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.226 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.226 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.226 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.226 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.226 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:58.228 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:33:58.228 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^18146\b' 2024-04-24T14:33:58.228 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^18146\b' 2024-04-24T14:33:58.229 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:33:58.229 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:58.231+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:58.230 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:58.230 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:33:58.231 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *18146\b' 2024-04-24T14:33:58.232 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *18146\b' 2024-04-24T14:33:58.240 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:58.241 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:33:58.329 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:33:58.331+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:33:58.336 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:58.336 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:58.336 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:58.336 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:58.345 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:33:58.429 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:58.429+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:58.448 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:58.448 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:58.448 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:58.448 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:58.448 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:58.457 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:33:58.530 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:58.529+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:58.563 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:379 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.902811891999999, 2024-04-24T14:33:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068078410000000002, 2024-04-24T14:33:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 379 2024-04-24T14:33:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:33:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664240+0000", 2024-04-24T14:33:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6539999999999999e-06 2024-04-24T14:33:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664245+0000", 2024-04-24T14:33:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2179999999999998e-06 2024-04-24T14:33:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664249+0000", 2024-04-24T14:33:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2279999999999998e-06 2024-04-24T14:33:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670245+0000", 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059962430000000001 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670400+0000", 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015535399999999999 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671046+0000", 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064591599999999996 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:380 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:33:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.902765264999999, 2024-04-24T14:33:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076160830000000001, 2024-04-24T14:33:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 380 2024-04-24T14:33:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664284+0000", 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664287+0000", 2024-04-24T14:33:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9739999999999998e-06 2024-04-24T14:33:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664302+0000", 2024-04-24T14:33:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5214999999999999e-05 2024-04-24T14:33:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664307+0000", 2024-04-24T14:33:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6859999999999997e-06 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671098+0000", 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067912479999999997 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671253+0000", 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015427400000000001 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671901+0000", 2024-04-24T14:33:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064845600000000005 2024-04-24T14:33:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.902735719999999, 2024-04-24T14:33:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:33:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:33:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:33:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:33:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:33:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:33:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:33:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:33:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:33:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:382 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:33:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.899734237000001, 2024-04-24T14:33:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067029910000000002, 2024-04-24T14:33:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 382 2024-04-24T14:33:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:33:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:33:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667315+0000", 2024-04-24T14:33:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667321+0000", 2024-04-24T14:33:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2099999999999998e-06 2024-04-24T14:33:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667336+0000", 2024-04-24T14:33:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4612e-05 2024-04-24T14:33:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667343+0000", 2024-04-24T14:33:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0119999999999999e-06 2024-04-24T14:33:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672957+0000", 2024-04-24T14:33:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056139700000000002 2024-04-24T14:33:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.673127+0000", 2024-04-24T14:33:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017044599999999999 2024-04-24T14:33:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674019+0000", 2024-04-24T14:33:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00089165499999999999 2024-04-24T14:33:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.899701193999999, 2024-04-24T14:33:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:33:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:33:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:33:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:33:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:33:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:33:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:33:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:33:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:33:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:33:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.899677008000001, 2024-04-24T14:33:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:33:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:33:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:33:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:33:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:33:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:33:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:33:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:33:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:33:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:33:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:33:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.899659624000002, 2024-04-24T14:33:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:33:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:33:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:33:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:33:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:33:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:33:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:33:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:33:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:399 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:33:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.885249489, 2024-04-24T14:33:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071317810000000002, 2024-04-24T14:33:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 399 2024-04-24T14:33:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:33:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:33:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681800+0000", 2024-04-24T14:33:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681804+0000", 2024-04-24T14:33:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.715e-06 2024-04-24T14:33:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681816+0000", 2024-04-24T14:33:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1851000000000001e-05 2024-04-24T14:33:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681820+0000", 2024-04-24T14:33:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6450000000000004e-06 2024-04-24T14:33:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688059+0000", 2024-04-24T14:33:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062390090000000002 2024-04-24T14:33:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688196+0000", 2024-04-24T14:33:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013721999999999999 2024-04-24T14:33:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688933+0000", 2024-04-24T14:33:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073605000000000003 2024-04-24T14:33:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:33:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.885223532000001, 2024-04-24T14:33:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:33:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:33:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:33:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:33:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:33:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:33:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:33:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:33:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:33:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:33:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:33:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:33:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.885191233, 2024-04-24T14:33:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:33:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:33:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:33:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:33:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:33:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:33:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:33:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:33:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:33:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:414 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.870995823000001, 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069877910000000001, 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 414 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:33:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696053+0000", 2024-04-24T14:33:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696058+0000", 2024-04-24T14:33:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7509999999999999e-06 2024-04-24T14:33:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696074+0000", 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5743000000000002e-05 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696093+0000", 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9508e-05 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702314+0000", 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062207010000000004 2024-04-24T14:33:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702431+0000", 2024-04-24T14:33:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011723299999999999 2024-04-24T14:33:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703042+0000", 2024-04-24T14:33:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000610752 2024-04-24T14:33:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.870938956, 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:33:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:33:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:33:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:33:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:33:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:33:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:33:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:33:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:33:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:33:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:33:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:33:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.870909856000001, 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:33:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:33:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:33:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:33:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:33:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:33:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:33:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:33:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:33:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:33:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:33:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:33:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:33:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.870883926000001, 2024-04-24T14:33:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:33:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:33:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:33:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:33:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:33:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:33:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:429 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3538944~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.856602486, 2024-04-24T14:33:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071866789999999996, 2024-04-24T14:33:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 429 2024-04-24T14:33:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:33:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:33:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710447+0000", 2024-04-24T14:33:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710451+0000", 2024-04-24T14:33:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5930000000000002e-06 2024-04-24T14:33:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710465+0000", 2024-04-24T14:33:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.411e-05 2024-04-24T14:33:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710470+0000", 2024-04-24T14:33:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3129999999999998e-06 2024-04-24T14:33:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716696+0000", 2024-04-24T14:33:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006225983 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716901+0000", 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020550200000000001 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717634+0000", 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073292099999999998 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:430 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:33:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.855238178, 2024-04-24T14:33:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068326970000000004, 2024-04-24T14:33:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 430 2024-04-24T14:33:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711810+0000", 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711822+0000", 2024-04-24T14:33:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2166e-05 2024-04-24T14:33:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711856+0000", 2024-04-24T14:33:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3816000000000003e-05 2024-04-24T14:33:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711873+0000", 2024-04-24T14:33:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6742999999999999e-05 2024-04-24T14:33:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717693+0000", 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058204579999999997 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717864+0000", 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017128299999999999 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718645+0000", 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078033699999999998 2024-04-24T14:33:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.855163462, 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:33:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:33:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:33:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:33:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:33:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:33:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:33:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:33:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:33:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:33:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:33:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:33:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:33:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:33:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.855131530000001, 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:33:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:33:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:33:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:33:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:33:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:33:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:33:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:33:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:33:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:33:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:33:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:33:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.855064014, 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:33:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:33:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:33:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:33:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:33:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:33:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:33:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.797651130999999, 2024-04-24T14:33:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:33:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:33:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:33:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:33:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:33:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:33:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:33:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:33:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:33:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:58.648 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:58.650 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:33:58.650 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:58.653+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:58.742 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:33:58.742 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd snap create rbd/ns/testrbdnbd16809@snap 2024-04-24T14:33:58.750 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:58.749+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:58.753 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:33:58.754 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:58.762 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:33:58.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T14:33:58.851 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:58.853+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:58.856 INFO:tasks.workunit.client.0.smithi173.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-04-24T14:33:58.871 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd16809@snap 2024-04-24T14:33:58.871 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:33:58.872 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:33:58.874 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:33:58.874 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:33:58.877 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:33:58.877 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:33:58.877 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd16809@snap 2024-04-24T14:33:58.911 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:58.911 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:33:58.911 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:33:58.911 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:33:58.911 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:58.920 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:33:58.951 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:58.953+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:59.020 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:33:59.021 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd ns 2024-04-24T14:33:59.021 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:33:59.021 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns=ns 2024-04-24T14:33:59.021 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:33:59.022 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:33:59.025 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:33:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:33:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:33:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:33:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:1 3.2 3:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:33:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.826888367000002, 2024-04-24T14:33:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002159392, 2024-04-24T14:33:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:33:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:33:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:33:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:33:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 1 2024-04-24T14:33:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201714+0000", 2024-04-24T14:33:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201710+0000", 2024-04-24T14:33:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:33:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201731+0000", 2024-04-24T14:33:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0835999999999998e-05 2024-04-24T14:33:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201769+0000", 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7472999999999998e-05 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201796+0000", 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6798999999999999e-05 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.201923+0000", 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012770699999999999 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:13.203873+0000", 2024-04-24T14:33:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00194992 2024-04-24T14:33:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:33:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.956670383999999, 2024-04-24T14:33:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:33:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:33:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:33:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:33:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:33:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:33:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:33:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:33:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:33:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:33:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:33:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:33:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:33:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:52 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'51, mlcod=57'49)", 2024-04-24T14:33:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.061270+0000", 2024-04-24T14:33:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.9673320959999998, 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024587599999999999, 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061270+0000", 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061270+0000", 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061268+0000", 2024-04-24T14:33:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:33:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061425+0000", 2024-04-24T14:33:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015709400000000001 2024-04-24T14:33:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061469+0000", 2024-04-24T14:33:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4759000000000003e-05 2024-04-24T14:33:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061488+0000", 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.821e-05 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061634+0000", 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014643300000000001 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.061712+0000", 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8019e-05 2024-04-24T14:33:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.063621+0000", 2024-04-24T14:33:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001909004 2024-04-24T14:33:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.063729+0000", 2024-04-24T14:33:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107685 2024-04-24T14:33:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:33:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:33:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.9618558049999999, 2024-04-24T14:33:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:33:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:33:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:33:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:33:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:33:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:33:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:33:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:33:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:33:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:33:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:33:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:33:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:33:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:33:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:33:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:33:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.9509265469999999, 2024-04-24T14:33:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:33:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:33:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:33:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:33:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:33:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:33:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:33:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:33:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:33:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:33:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:33:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:33:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:33:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:33:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:33:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.9399704660000001, 2024-04-24T14:33:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:33:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:33:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:33:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:33:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:33:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:33:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:33:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:33:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:33:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:33:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:33:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:33:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:33:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:33:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:33:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.9284011010000004, 2024-04-24T14:33:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:33:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:33:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:33:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:33:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:33:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:33:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:33:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:33:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:33:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:33:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:33:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:33:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:33:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:109 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'60, mlcod=62'58)", 2024-04-24T14:33:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:33:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.6852889360000001, 2024-04-24T14:33:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036290020000000001, 2024-04-24T14:33:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:33:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:33:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343311+0000", 2024-04-24T14:33:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343841+0000", 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053003399999999997 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343876+0000", 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4479e-05 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343888+0000", 2024-04-24T14:33:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1623e-05 2024-04-24T14:33:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343996+0000", 2024-04-24T14:33:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108028 2024-04-24T14:33:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.344044+0000", 2024-04-24T14:33:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8081999999999998e-05 2024-04-24T14:33:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.346872+0000", 2024-04-24T14:33:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028280789999999998 2024-04-24T14:33:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.346942+0000", 2024-04-24T14:33:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0405000000000001e-05 2024-04-24T14:33:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:33:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:33:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.6752814420000002, 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:33:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:33:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:33:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:33:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:33:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:33:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:33:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:33:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:33:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:33:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:33:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:33:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:33:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:33:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:33:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:33:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:33:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.6645362739999996, 2024-04-24T14:33:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:33:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:33:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:33:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:33:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:33:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:33:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:33:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:33:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:33:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:33:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:33:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:33:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:33:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:33:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:33:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:33:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.6521699979999998, 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:33:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:33:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:33:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:33:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:33:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:33:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:33:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:33:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:33:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:33:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:33:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:33:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:33:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:33:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:33:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:33:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:33:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:33:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:33:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.5272338770000005, 2024-04-24T14:33:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:33:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:33:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:33:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:33:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:33:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:33:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:33:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:33:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:33:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:33:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:33:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:33:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:138 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'70, mlcod=62'68)", 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.5168477229999997, 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036613330000000001, 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:33:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:33:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511753+0000", 2024-04-24T14:33:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512370+0000", 2024-04-24T14:33:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061776500000000005 2024-04-24T14:33:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512395+0000", 2024-04-24T14:33:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4549000000000002e-05 2024-04-24T14:33:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512404+0000", 2024-04-24T14:33:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1130000000000007e-06 2024-04-24T14:33:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512517+0000", 2024-04-24T14:33:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112679 2024-04-24T14:33:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512581+0000", 2024-04-24T14:33:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4274000000000002e-05 2024-04-24T14:33:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.515303+0000", 2024-04-24T14:33:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027220640000000002 2024-04-24T14:33:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.515416+0000", 2024-04-24T14:33:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112644 2024-04-24T14:33:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:33:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:33:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.5078460339999999, 2024-04-24T14:33:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:33:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:33:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:33:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:33:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:33:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:33:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:33:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:33:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:33:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:33:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:33:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:33:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:33:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:33:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.4981181909999997, 2024-04-24T14:33:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:33:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:33:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:33:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:33:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:33:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:33:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:33:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:33:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:33:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:33:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:33:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:33:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:33:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:33:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:33:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:33:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:33:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:33:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:171 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'75, mlcod=62'73)", 2024-04-24T14:33:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.675580+0000", 2024-04-24T14:33:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.3530223499999998, 2024-04-24T14:33:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.002272273, 2024-04-24T14:33:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675580+0000", 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675580+0000", 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675578+0000", 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675717+0000", 2024-04-24T14:33:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013899999999999999 2024-04-24T14:33:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675758+0000", 2024-04-24T14:33:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1696999999999997e-05 2024-04-24T14:33:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675775+0000", 2024-04-24T14:33:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6430999999999999e-05 2024-04-24T14:33:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675878+0000", 2024-04-24T14:33:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000102749 2024-04-24T14:33:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.675940+0000", 2024-04-24T14:33:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2794999999999998e-05 2024-04-24T14:33:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.677755+0000", 2024-04-24T14:33:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0018146060000000001 2024-04-24T14:33:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.677852+0000", 2024-04-24T14:33:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7054999999999998e-05 2024-04-24T14:33:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:172 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'76, mlcod=62'74)", 2024-04-24T14:33:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:33:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.3478508370000002, 2024-04-24T14:33:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032080400000000001, 2024-04-24T14:33:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:33:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:33:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680750+0000", 2024-04-24T14:33:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681277+0000", 2024-04-24T14:33:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052776400000000001 2024-04-24T14:33:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681304+0000", 2024-04-24T14:33:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6948e-05 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681315+0000", 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0648e-05 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681413+0000", 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7926999999999999e-05 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681470+0000", 2024-04-24T14:33:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7401999999999997e-05 2024-04-24T14:33:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.683923+0000", 2024-04-24T14:33:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024526790000000001 2024-04-24T14:33:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.683959+0000", 2024-04-24T14:33:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6327e-05 2024-04-24T14:33:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.338008511, 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:33:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:33:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:33:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:33:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:33:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:33:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:33:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:33:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:33:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:33:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:33:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:33:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:33:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:33:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:33:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:33:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:33:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:33:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.3282795150000002, 2024-04-24T14:33:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:33:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:33:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:33:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:33:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:33:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:33:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:33:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:33:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:33:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:33:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:33:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:33:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:33:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:178 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'82, mlcod=62'80)", 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.3178121389999999, 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029527640000000001, 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:33:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:33:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:33:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:33:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:33:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:33:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:33:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:33:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710789+0000", 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711538+0000", 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00074893099999999999 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711565+0000", 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7526999999999999e-05 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711576+0000", 2024-04-24T14:33:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0553999999999999e-05 2024-04-24T14:33:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:33:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711633+0000", 2024-04-24T14:33:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7258e-05 2024-04-24T14:33:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:33:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711653+0000", 2024-04-24T14:33:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0460999999999999e-05 2024-04-24T14:33:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:33:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.713712+0000", 2024-04-24T14:33:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020587790000000002 2024-04-24T14:33:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:33:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:33:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:33:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.713743+0000", 2024-04-24T14:33:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0519999999999999e-05 2024-04-24T14:33:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:33:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:33:59.111 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:33:59.114 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=18245 2024-04-24T14:33:59.114 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 18245 2024-04-24T14:33:59.114 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 18245 -C rbd-nbd 2024-04-24T14:33:59.115 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:33:59.115 INFO:tasks.workunit.client.0.smithi173.stdout: 18245 ? 00:00:00 rbd-nbd 2024-04-24T14:33:59.115 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:59.117+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:59.116 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:33:59.117 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device rbd/ns/testrbdnbd16809@snap 18245 2024-04-24T14:33:59.117 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=rbd/ns/testrbdnbd16809@snap 2024-04-24T14:33:59.117 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=18245 2024-04-24T14:33:59.117 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd16809@snap 2024-04-24T14:33:59.117 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:33:59.117 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:33:59.117 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:33:59.118 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:33:59.118 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:33:59.118 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:33:59.118 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd16809@snap 2024-04-24T14:33:59.179 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:33:59.179 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^18245\b' 2024-04-24T14:33:59.180 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^18245\b' 2024-04-24T14:33:59.215 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:59.217+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:59.227 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:59.227 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:59.227 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:59.227 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:59.234 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:59.235 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:33:59.236 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *18245\b' 2024-04-24T14:33:59.236 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *18245\b' 2024-04-24T14:33:59.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:33:59.238 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:33:59.239+0000 7f229e4073c0 -1 osd.1 31 log_to_monitors true 2024-04-24T14:33:59.249 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:33:59.249 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:33:59.317 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:33:59.317+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:33:59.324 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:59.324 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:59.324 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:59.324 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:59.324 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:59.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:33:59.416 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:59.419+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:59.431 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:59.431 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:59.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:59.431 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:59.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.431 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:97 3.1 3:990e5750:::rbd_header.12ff64619e2d:head [watch reconnect cookie 140276853203408 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:59.432 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:33:59.432 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.081325434, 2024-04-24T14:33:59.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012161632, 2024-04-24T14:33:59.432 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.432 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.432 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.432 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.432 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.432 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 97 2024-04-24T14:33:59.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.432 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:33:59.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:33:59.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352830+0000", 2024-04-24T14:33:59.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352834+0000", 2024-04-24T14:33:59.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2699999999999998e-06 2024-04-24T14:33:59.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352844+0000", 2024-04-24T14:33:59.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0406e-05 2024-04-24T14:33:59.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352850+0000", 2024-04-24T14:33:59.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6439999999999997e-06 2024-04-24T14:33:59.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362391+0000", 2024-04-24T14:33:59.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095408969999999996 2024-04-24T14:33:59.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362683+0000", 2024-04-24T14:33:59.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029238000000000002 2024-04-24T14:33:59.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.364992+0000", 2024-04-24T14:33:59.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023092020000000002 2024-04-24T14:33:59.436 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.436 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.436 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.436 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:225 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:59.436 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:59.436 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.928883322000001, 2024-04-24T14:33:59.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010959015000000001, 2024-04-24T14:33:59.436 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.436 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.436 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.437 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.437 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.437 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 225 2024-04-24T14:33:59.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.437 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:59.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:59.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:33:59.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:59.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505275+0000", 2024-04-24T14:33:59.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0269999999999998e-06 2024-04-24T14:33:59.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505279+0000", 2024-04-24T14:33:59.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4279999999999998e-06 2024-04-24T14:33:59.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505282+0000", 2024-04-24T14:33:59.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.982e-06 2024-04-24T14:33:59.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515209+0000", 2024-04-24T14:33:59.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099266419999999994 2024-04-24T14:33:59.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515466+0000", 2024-04-24T14:33:59.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025719700000000002 2024-04-24T14:33:59.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.516232+0000", 2024-04-24T14:33:59.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076608899999999996 2024-04-24T14:33:59.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.442 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.443 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.443 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:59.443 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.677851579999999, 2024-04-24T14:33:59.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:33:59.443 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.443 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.443 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.443 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.443 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.443 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:33:59.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.443 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:33:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:33:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:33:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:33:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:33:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:33:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:33:59.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:33:59.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:33:59.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:33:59.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:33:59.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:33:59.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:33:59.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:33:59.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:33:59.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:33:59.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:33:59.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:33:59.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:33:59.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:33:59.448 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.448 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.448 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.448 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.448 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:59.448 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.677800745999999, 2024-04-24T14:33:59.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:33:59.449 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.449 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.449 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.449 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.449 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.449 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:33:59.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.449 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:59.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:59.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:33:59.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:59.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:33:59.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:33:59.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:33:59.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:33:59.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:33:59.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:33:59.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:33:59.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:33:59.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:33:59.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:33:59.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:33:59.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:33:59.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:33:59.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:33:59.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:33:59.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:33:59.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:33:59.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:33:59.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:33:59.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:33:59.454 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.454 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.454 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.454 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.677764439000001, 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:59.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:33:59.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:33:59.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:59.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:33:59.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:33:59.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:33:59.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:33:59.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:33:59.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:33:59.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:33:59.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:33:59.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:33:59.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:33:59.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:33:59.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:33:59.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:33:59.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:33:59.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:33:59.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:33:59.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:33:59.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:33:59.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:33:59.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:33:59.460 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.460 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.460 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.461 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.461 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:59.461 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.674353189000001, 2024-04-24T14:33:59.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:33:59.461 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.461 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.461 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.461 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.461 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.461 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:33:59.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.461 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:59.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:33:59.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:33:59.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:59.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:33:59.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:33:59.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:33:59.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:33:59.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:33:59.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:33:59.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:33:59.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:33:59.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:33:59.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:33:59.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:33:59.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:33:59.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:33:59.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:33:59.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:33:59.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:33:59.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:33:59.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:33:59.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:33:59.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:33:59.467 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.467 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.467 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.467 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.467 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:59.468 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.674291512, 2024-04-24T14:33:59.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:33:59.468 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.468 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.468 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.468 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.468 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.468 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:33:59.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.468 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:59.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:33:59.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:33:59.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:33:59.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:33:59.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:33:59.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:33:59.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:33:59.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:33:59.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:33:59.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:33:59.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:33:59.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:33:59.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:33:59.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:33:59.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:33:59.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:33:59.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:33:59.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:33:59.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:33:59.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:33:59.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:33:59.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:33:59.473 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.473 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.473 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.473 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.474 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:59.474 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.674256935999999, 2024-04-24T14:33:59.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:33:59.474 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.474 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.474 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.474 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.474 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.474 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:33:59.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.474 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:59.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:59.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:33:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:33:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:33:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:33:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:33:59.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:33:59.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:33:59.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:33:59.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:33:59.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:33:59.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:33:59.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:33:59.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:33:59.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:33:59.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:33:59.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:33:59.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:33:59.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:33:59.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:33:59.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:33:59.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:33:59.480 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.480 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.480 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.480 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.480 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:59.480 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.674227597000002, 2024-04-24T14:33:59.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:33:59.480 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.481 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.481 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:33:59.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.481 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:59.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:33:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:33:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:33:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:33:59.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:33:59.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:33:59.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:33:59.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:33:59.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:33:59.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:33:59.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:33:59.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:33:59.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:33:59.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:33:59.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:33:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:33:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:33:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:33:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:33:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:33:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:33:59.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:33:59.486 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.486 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.486 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.486 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.486 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:59.486 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.674151015, 2024-04-24T14:33:59.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:33:59.486 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.486 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.486 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.487 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.487 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.487 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:33:59.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.487 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:59.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:33:59.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:33:59.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:33:59.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:33:59.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:33:59.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:33:59.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:33:59.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:33:59.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:33:59.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:33:59.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:33:59.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:33:59.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:33:59.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:33:59.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:33:59.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:33:59.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:33:59.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:33:59.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:33:59.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:33:59.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:33:59.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:33:59.493 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.493 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.493 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.493 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.493 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:59.493 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.674103469999999, 2024-04-24T14:33:59.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:33:59.493 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.493 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.494 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.494 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.494 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.494 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:33:59.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.494 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:59.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:33:59.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:33:59.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:33:59.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:33:59.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:33:59.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:33:59.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:33:59.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:33:59.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:33:59.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:33:59.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:33:59.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:33:59.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:33:59.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:33:59.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:33:59.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:33:59.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:33:59.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:33:59.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:33:59.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:33:59.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:33:59.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:33:59.499 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.499 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.499 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.499 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.499 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.674062828, 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:59.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:33:59.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:59.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:33:59.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:33:59.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:33:59.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:33:59.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:33:59.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:33:59.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:33:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:33:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:33:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:33:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:33:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:33:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:33:59.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:33:59.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:33:59.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:33:59.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:33:59.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:33:59.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:33:59.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:33:59.505 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.505 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.505 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.505 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.506 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:59.506 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.674045716999998, 2024-04-24T14:33:59.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:33:59.506 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.506 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.506 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.506 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.507 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.507 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:33:59.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.507 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:59.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:33:59.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:33:59.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:59.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:33:59.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:33:59.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:33:59.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:33:59.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:33:59.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:33:59.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:33:59.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:33:59.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:33:59.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:33:59.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:33:59.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:33:59.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:33:59.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:33:59.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:33:59.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:33:59.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:33:59.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:33:59.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:33:59.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:33:59.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.515 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.515 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:481 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.516 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:33:59.516 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.640174009999999, 2024-04-24T14:33:59.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010661718000000001, 2024-04-24T14:33:59.516 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.516 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.517 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.517 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.517 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.517 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 481 2024-04-24T14:33:59.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.517 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:33:59.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:33:59.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793981+0000", 2024-04-24T14:33:59.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793985+0000", 2024-04-24T14:33:59.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8469999999999998e-06 2024-04-24T14:33:59.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:33:59.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0220000000000004e-06 2024-04-24T14:33:59.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793998+0000", 2024-04-24T14:33:59.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8559999999999998e-06 2024-04-24T14:33:59.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803807+0000", 2024-04-24T14:33:59.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098088629999999993 2024-04-24T14:33:59.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803916+0000", 2024-04-24T14:33:59.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010854899999999999 2024-04-24T14:33:59.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.804644+0000", 2024-04-24T14:33:59.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072835399999999996 2024-04-24T14:33:59.523 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.523 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.523 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.523 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:497 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.524 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:33:59.524 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.622233697999999, 2024-04-24T14:33:59.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013996471, 2024-04-24T14:33:59.524 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.524 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.524 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.524 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.524 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.524 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 497 2024-04-24T14:33:59.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.524 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:33:59.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:33:59.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811922+0000", 2024-04-24T14:33:59.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:59.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811925+0000", 2024-04-24T14:33:59.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4120000000000001e-06 2024-04-24T14:33:59.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811936+0000", 2024-04-24T14:33:59.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1077000000000001e-05 2024-04-24T14:33:59.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811941+0000", 2024-04-24T14:33:59.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7539999999999997e-06 2024-04-24T14:33:59.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822289+0000", 2024-04-24T14:33:59.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010347635000000001 2024-04-24T14:33:59.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822457+0000", 2024-04-24T14:33:59.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000168257 2024-04-24T14:33:59.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.825919+0000", 2024-04-24T14:33:59.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003461979 2024-04-24T14:33:59.530 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.530 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.530 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.531 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:25 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:33:59.531 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:33:59.531 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.466675757, 2024-04-24T14:33:59.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013349789000000001, 2024-04-24T14:33:59.531 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.531 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.531 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.531 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:33:59.531 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:33:59.532 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 25 2024-04-24T14:33:59.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.532 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:33:59.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:33:59.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967479+0000", 2024-04-24T14:33:59.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967669+0000", 2024-04-24T14:33:59.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000189829 2024-04-24T14:33:59.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967687+0000", 2024-04-24T14:33:59.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8199999999999999e-05 2024-04-24T14:33:59.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967695+0000", 2024-04-24T14:33:59.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4869999999999996e-06 2024-04-24T14:33:59.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.972501+0000", 2024-04-24T14:33:59.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048057509999999996 2024-04-24T14:33:59.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.972824+0000", 2024-04-24T14:33:59.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032322299999999998 2024-04-24T14:33:59.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:33:59.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.973071+0000", 2024-04-24T14:33:59.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024751299999999999 2024-04-24T14:33:59.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:33:59.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977624+0000", 2024-04-24T14:33:59.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045527570000000002 2024-04-24T14:33:59.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:59.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980627+0000", 2024-04-24T14:33:59.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003002972 2024-04-24T14:33:59.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:59.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980743+0000", 2024-04-24T14:33:59.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2024-04-24T14:33:59.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980830+0000", 2024-04-24T14:33:59.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7534999999999995e-05 2024-04-24T14:33:59.540 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.540 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.540 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.541 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:26 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:33:59.541 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:33:59.541 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.4664520220000004, 2024-04-24T14:33:59.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013403597999999999, 2024-04-24T14:33:59.541 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.542 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.542 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.542 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:33:59.542 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:33:59.542 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 26 2024-04-24T14:33:59.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.543 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:33:59.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:33:59.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967703+0000", 2024-04-24T14:33:59.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967874+0000", 2024-04-24T14:33:59.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017088499999999999 2024-04-24T14:33:59.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967886+0000", 2024-04-24T14:33:59.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1744e-05 2024-04-24T14:33:59.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967893+0000", 2024-04-24T14:33:59.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2024-04-24T14:33:59.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974028+0000", 2024-04-24T14:33:59.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061356650000000002 2024-04-24T14:33:59.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974347+0000", 2024-04-24T14:33:59.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031897400000000003 2024-04-24T14:33:59.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:33:59.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974774+0000", 2024-04-24T14:33:59.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042661700000000001 2024-04-24T14:33:59.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:59.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980930+0000", 2024-04-24T14:33:59.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061563360000000001 2024-04-24T14:33:59.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:33:59.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980945+0000", 2024-04-24T14:33:59.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4864e-05 2024-04-24T14:33:59.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:59.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981035+0000", 2024-04-24T14:33:59.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9390999999999998e-05 2024-04-24T14:33:59.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981108+0000", 2024-04-24T14:33:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3255000000000003e-05 2024-04-24T14:33:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:27 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:33:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:33:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.4662548759999998, 2024-04-24T14:33:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013394262000000001, 2024-04-24T14:33:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:33:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:33:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 27 2024-04-24T14:33:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:33:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:33:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967900+0000", 2024-04-24T14:33:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968061+0000", 2024-04-24T14:33:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016101900000000001 2024-04-24T14:33:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968072+0000", 2024-04-24T14:33:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1012999999999999e-05 2024-04-24T14:33:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968079+0000", 2024-04-24T14:33:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.46e-06 2024-04-24T14:33:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.976820+0000", 2024-04-24T14:33:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087406289999999998 2024-04-24T14:33:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.976951+0000", 2024-04-24T14:33:59.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131187 2024-04-24T14:33:59.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:33:59.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977129+0000", 2024-04-24T14:33:59.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000178135 2024-04-24T14:33:59.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:33:59.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981126+0000", 2024-04-24T14:33:59.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00399709 2024-04-24T14:33:59.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:59.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981193+0000", 2024-04-24T14:33:59.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7083999999999997e-05 2024-04-24T14:33:59.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981251+0000", 2024-04-24T14:33:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7754000000000003e-05 2024-04-24T14:33:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981296+0000", 2024-04-24T14:33:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4765000000000003e-05 2024-04-24T14:33:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:28 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:33:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:33:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.4660696440000001, 2024-04-24T14:33:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013427247999999999, 2024-04-24T14:33:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:33:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:33:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 28 2024-04-24T14:33:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:33:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:33:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968086+0000", 2024-04-24T14:33:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968247+0000", 2024-04-24T14:33:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016121199999999999 2024-04-24T14:33:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968260+0000", 2024-04-24T14:33:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.275e-05 2024-04-24T14:33:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968266+0000", 2024-04-24T14:33:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3770000000000001e-06 2024-04-24T14:33:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977699+0000", 2024-04-24T14:33:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094331439999999992 2024-04-24T14:33:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977894+0000", 2024-04-24T14:33:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194945 2024-04-24T14:33:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:33:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.978258+0000", 2024-04-24T14:33:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036359400000000002 2024-04-24T14:33:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:33:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981140+0000", 2024-04-24T14:33:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028821540000000001 2024-04-24T14:33:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981360+0000", 2024-04-24T14:33:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022056300000000001 2024-04-24T14:33:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981444+0000", 2024-04-24T14:33:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3733999999999995e-05 2024-04-24T14:33:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981514+0000", 2024-04-24T14:33:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9702000000000005e-05 2024-04-24T14:33:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:29 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:33:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:33:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.4658827990000001, 2024-04-24T14:33:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014492283, 2024-04-24T14:33:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:33:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:33:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:33:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 29 2024-04-24T14:33:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:33:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:33:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:33:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968434+0000", 2024-04-24T14:33:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016130199999999999 2024-04-24T14:33:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968484+0000", 2024-04-24T14:33:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0003000000000002e-05 2024-04-24T14:33:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968499+0000", 2024-04-24T14:33:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4628e-05 2024-04-24T14:33:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979299+0000", 2024-04-24T14:33:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010800624999999999 2024-04-24T14:33:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979488+0000", 2024-04-24T14:33:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000189037 2024-04-24T14:33:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:33:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979784+0000", 2024-04-24T14:33:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029562000000000002 2024-04-24T14:33:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:33:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982305+0000", 2024-04-24T14:33:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025212339999999998 2024-04-24T14:33:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:33:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982680+0000", 2024-04-24T14:33:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000374762 2024-04-24T14:33:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:33:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982731+0000", 2024-04-24T14:33:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1146999999999997e-05 2024-04-24T14:33:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982766+0000", 2024-04-24T14:33:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4755000000000002e-05 2024-04-24T14:33:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.583 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.583 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:59.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:33:59.585 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:33:59.591+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:33:59.685 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:59.689+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:59.693 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:59.693 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:59.693 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:33:59.693 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:59.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:33:59.750 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:33:59.752 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd snap ls rbd/ns/testrbdnbd16809 2024-04-24T14:33:59.752 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '$2 == "snap" {print $1}' 2024-04-24T14:33:59.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:33:59.785 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:33:59.785+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:33:59.801 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:33:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:33:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:33:59.801 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:33:59.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:33:59.813 INFO:tasks.workunit.client.0.smithi173.stderr:+ SNAPID=5 2024-04-24T14:33:59.814 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map --snap-id 5 rbd/ns/testrbdnbd16809 2024-04-24T14:33:59.814 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:33:59.815 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:33:59.816 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:33:59.817 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:33:59.819 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:33:59.819 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:33:59.819 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 5 rbd/ns/testrbdnbd16809 2024-04-24T14:33:59.865 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-24T14:33:59.867 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:33:59.871+0000 7f229294a700 -1 osd.1 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T14:33:59.886 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:33:59.889+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:33:59.924 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:33:59.924 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:33:59.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:33:59.924 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:33:59.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.925 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:192 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:59.925 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:59.925 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.457511376999999, 2024-04-24T14:33:59.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010036329, 2024-04-24T14:33:59.925 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.925 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.925 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.925 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.925 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.925 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192 2024-04-24T14:33:59.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.925 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:59.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:33:59.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471099+0000", 2024-04-24T14:33:59.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471103+0000", 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8800000000000001e-06 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471111+0000", 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2930000000000001e-06 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471116+0000", 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1800000000000004e-06 2024-04-24T14:33:59.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480338+0000", 2024-04-24T14:33:59.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092213069999999998 2024-04-24T14:33:59.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480467+0000", 2024-04-24T14:33:59.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012901500000000001 2024-04-24T14:33:59.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481136+0000", 2024-04-24T14:33:59.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066948400000000003 2024-04-24T14:33:59.929 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.929 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.929 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.929 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:33:59.929 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:59.929 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.457488341000001, 2024-04-24T14:33:59.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:33:59.929 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.929 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.929 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.929 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.929 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.930 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:33:59.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.930 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:59.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:33:59.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:33:59.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:33:59.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:33:59.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:33:59.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:33:59.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:33:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:33:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:33:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:33:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:33:59.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:33:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:33:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:33:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.082131638, 2024-04-24T14:33:59.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:33:59.934 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.934 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.934 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.934 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.934 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.934 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:33:59.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.934 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:59.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:33:59.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:33:59.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:33:59.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:33:59.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:33:59.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:33:59.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:33:59.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:33:59.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:33:59.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:33:59.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:33:59.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:33:59.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:33:59.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:33:59.937 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.937 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.937 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.938 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:526 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.938 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:59.938 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.080744115000002, 2024-04-24T14:33:59.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010568783999999999, 2024-04-24T14:33:59.938 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.938 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.938 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 526 2024-04-24T14:33:59.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.938 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:59.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:33:59.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847866+0000", 2024-04-24T14:33:59.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847871+0000", 2024-04-24T14:33:59.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1590000000000001e-06 2024-04-24T14:33:59.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847883+0000", 2024-04-24T14:33:59.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.172e-05 2024-04-24T14:33:59.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847890+0000", 2024-04-24T14:33:59.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6950000000000003e-06 2024-04-24T14:33:59.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857418+0000", 2024-04-24T14:33:59.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095285909999999995 2024-04-24T14:33:59.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857588+0000", 2024-04-24T14:33:59.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169541 2024-04-24T14:33:59.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858436+0000", 2024-04-24T14:33:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084814700000000005 2024-04-24T14:33:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.080703484000001, 2024-04-24T14:33:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:33:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.943 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.943 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.943 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:33:59.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.943 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:59.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:33:59.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:33:59.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:33:59.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:33:59.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:33:59.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:33:59.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:33:59.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:33:59.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:33:59.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:33:59.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:33:59.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:33:59.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:33:59.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:33:59.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.946 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.946 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.080676395000001, 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:59.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:33:59.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:33:59.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:59.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:33:59.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:33:59.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:33:59.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:33:59.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:33:59.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:33:59.950 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.950 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.951 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.951 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.951 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:59.951 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.080652428, 2024-04-24T14:33:59.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:33:59.951 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.951 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.951 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.951 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.951 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.951 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:33:59.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.952 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:59.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:33:59.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:33:59.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:59.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:33:59.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:33:59.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:33:59.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:33:59.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:33:59.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:33:59.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:33:59.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:33:59.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:33:59.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:33:59.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:33:59.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:33:59.955 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.955 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.955 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.955 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.955 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:59.955 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.066668449000002, 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:33:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:33:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:33:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:33:59.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:33:59.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:33:59.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:33:59.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:33:59.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:33:59.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:33:59.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:33:59.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:33:59.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:33:59.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:33:59.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:33:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:33:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:33:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:33:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:33:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:33:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:33:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:33:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.066552137999999, 2024-04-24T14:33:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:33:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:33:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:33:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:33:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:33:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:33:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:33:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:33:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:33:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:33:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:33:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:33:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:33:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.066504479999999, 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:33:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:33:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:33:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:33:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:33:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:33:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:33:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:33:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:33:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:33:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:33:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:33:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:33:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:33:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:33:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:33:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:33:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:33:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:33:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:33:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:33:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:33:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:33:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.062467181999999, 2024-04-24T14:33:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:33:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:33:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:33:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:33:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:33:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:33:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:33:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:33:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:33:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:33:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:33:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:33:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:33:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:33:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:33:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:33:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:33:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:33:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:33:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:33:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:33:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:33:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:33:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:33:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:33:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.062403422999999, 2024-04-24T14:33:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:33:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:33:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:33:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:33:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:33:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:33:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:33:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:33:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:33:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:33:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:33:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:33:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:33:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:33:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:33:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:33:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:33:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:33:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.062377933, 2024-04-24T14:33:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:33:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:33:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:33:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:33:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:33:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:33:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:33:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:33:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:33:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:33:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:33:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:33:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:33:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:33:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:33:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:33:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:33:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.062354183, 2024-04-24T14:33:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:33:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:33:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:33:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:33:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:33:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:33:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:33:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:33:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:33:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:33:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:33:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:33:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:33:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:33:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:33:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:33:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:33:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:33:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:33:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:33:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:33:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:33:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:33:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:33:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:33:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:33:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.06231567, 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:33:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:33:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:33:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:33:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:33:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:33:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:33:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:33:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:33:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:33:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:33:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:33:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:33:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:33:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:33:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:33:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:33:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:33:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:33:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:33:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:33:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:33:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:33:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:33:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:33:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:33:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:33:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:34:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:34:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:34:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:34:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:34:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:34:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:34:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:34:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:34:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:34:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:34:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:34:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.062287473000001, 2024-04-24T14:34:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:34:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:34:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:34:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:34:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:34:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:34:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:34:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:34:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:34:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:34:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:34:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:34:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:34:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:34:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:34:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:34:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:34:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:34:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.062266015999999, 2024-04-24T14:34:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:34:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:34:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:34:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:34:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:34:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:34:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:34:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:34:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:34:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:34:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:34:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:34:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:34:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:34:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:34:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:34:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.062247173999999, 2024-04-24T14:34:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:34:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:34:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:34:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:34:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:34:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:34:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:34:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:34:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:34:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:34:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:34:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:34:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:34:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:34:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:34:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:34:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:34:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:34:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:34:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:34:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:34:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:34:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:34:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:561 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:00.020 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.020284554, 2024-04-24T14:34:00.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01066806, 2024-04-24T14:34:00.020 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:00.020 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:00.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:00.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:00.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:00.020 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 561 2024-04-24T14:34:00.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.020 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:00.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:00.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:00.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:00.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:00.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:00.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:00.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:00.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908326+0000", 2024-04-24T14:34:00.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:00.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:00.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908329+0000", 2024-04-24T14:34:00.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.802e-06 2024-04-24T14:34:00.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:00.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908338+0000", 2024-04-24T14:34:00.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6260000000000001e-06 2024-04-24T14:34:00.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:00.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908345+0000", 2024-04-24T14:34:00.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1389999999999998e-06 2024-04-24T14:34:00.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.917978+0000", 2024-04-24T14:34:00.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096329840000000007 2024-04-24T14:34:00.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:00.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918178+0000", 2024-04-24T14:34:00.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019999199999999999 2024-04-24T14:34:00.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:00.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918995+0000", 2024-04-24T14:34:00.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081708099999999999 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.84941502499999999, 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:00.024 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:34:00.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.025 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:00.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:00.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:00.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:00.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:00.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:00.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:00.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:00.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:34:00.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:00.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:00.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:34:00.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:34:00.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:00.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:34:00.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:00.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:00.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:34:00.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:34:00.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:34:00.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:34:00.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:00.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:34:00.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:34:00.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:34:00.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:34:00.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:00.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:34:00.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:34:00.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:00.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:34:00.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:34:00.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:00.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:34:00.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:34:00.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:00.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:00.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:00.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:34:00.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:34:00.029 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:00.029 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:00.029 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:00.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:00.030 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:00.030 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:00.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:00.031 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:00.032 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:34:00.032 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd ns 2024-04-24T14:34:00.032 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:00.032 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns=ns 2024-04-24T14:34:00.032 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:00.033 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:00.033 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:00.034 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:00.034 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:00.034 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:00.034 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:00.035 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-24T14:34:00.035 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:00.035 INFO:teuthology.orchestra.run.smithi090.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-24T14:34:00.035 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:00.036 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:00.036 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:00.036 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:00.036 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-24T14:34:00.036 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:00.035+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:00.075 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=18331 2024-04-24T14:34:00.075 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 18331 2024-04-24T14:34:00.075 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 18331 -C rbd-nbd 2024-04-24T14:34:00.084 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:00.084 INFO:tasks.workunit.client.0.smithi173.stdout: 18331 ? 00:00:00 rbd-nbd 2024-04-24T14:34:00.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device '--snap-id 5 rbd/ns/testrbdnbd16809' 18331 2024-04-24T14:34:00.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ local 'args=--snap-id 5 rbd/ns/testrbdnbd16809' 2024-04-24T14:34:00.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=18331 2024-04-24T14:34:00.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 5 rbd/ns/testrbdnbd16809 2024-04-24T14:34:00.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:00.086 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:00.087 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:00.087 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:00.088 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:00.088 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:00.088 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 5 rbd/ns/testrbdnbd16809 2024-04-24T14:34:00.133 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:00.135+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:00.143 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:00.143 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:00.152 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:00.170 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:34:00.170 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^18331\b' 2024-04-24T14:34:00.170 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^18331\b' 2024-04-24T14:34:00.173 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:00.173 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:00.222 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:00.223 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:34:00.223 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *18331\b' 2024-04-24T14:34:00.223 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *18331\b' 2024-04-24T14:34:00.229 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:00.229 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:34:00.233 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:00.239+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:00.253 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:00.253 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:00.262 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:00.334 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:00.339+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:00.360 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:00.360 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:00.369 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:00.435 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:00.437+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:00.467 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:00.467 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:00.476 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:00.536 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:00.541+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:00.620 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:00.620 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:00.630 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:00.675 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:00.681+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:00.730 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:00.730 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:34:00.731 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd16809 2024-04-24T14:34:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:34:00.732 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:34:00.734 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:34:00.735 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:34:00.737 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:34:00.737 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:34:00.737 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd16809 2024-04-24T14:34:00.738 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:00.737+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:00.739 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:00.838 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:00.837+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:00.840 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:00.840 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:00.849 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:34:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd ns 2024-04-24T14:34:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns=ns 2024-04-24T14:34:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:00.893 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:00.939 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:00.943+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=18395 2024-04-24T14:34:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 18395 2024-04-24T14:34:00.963 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 18395 -C rbd-nbd 2024-04-24T14:34:00.975 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:00.975 INFO:tasks.workunit.client.0.smithi173.stdout: 18395 ? 00:00:00 rbd-nbd 2024-04-24T14:34:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd16809' 18395 2024-04-24T14:34:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd16809' 2024-04-24T14:34:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=18395 2024-04-24T14:34:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd16809 2024-04-24T14:34:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:00.976 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:00.977 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:00.978 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:00.979 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:00.979 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:00.979 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd16809 2024-04-24T14:34:00.991 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:00.991 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:01.000 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:01.039 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:01.043+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:01.101 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:01.101 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:01.110 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:01.140 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:01.145+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:01.223 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:01.223 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:01.233 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:01.241 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:01.245+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:01.332 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:01.332 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:01.332 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:01.332 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:01.332 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:01.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:01.342 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:01.345+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:01.441 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:01.441 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:01.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:01.441 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [] 2024-04-24T14:34:01.442 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:01.442 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:01.445+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:01.450 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:01.543 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:01.547+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:01.551 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:01.552 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:01.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:01.643 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:01.647+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:01.701 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:01.701 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:01.709 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:01.754 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:01.759+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:01.806 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:01.807 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:01.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:01.846 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:01.851+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:01.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T14:34:01.936 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:01.937 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:01.947 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:01.953+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:01.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:02.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:34:02.047 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:02.051+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:02.059 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:02.059 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:02.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:02.148 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:02.153+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:02.168 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:02.172 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:02.172 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:02.181 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:02.186 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:02.206 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:02.232 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:02.249 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:02.253+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:02.253 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:02.275 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:02.278 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:02.278 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-04-24T14:34:02.278 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:02.278 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:02.279 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:02.279 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:02.290 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:02.290 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:02.291 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:02.291 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:02.300 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:02.350 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:02.355+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:02.406 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:02.406 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:02.406 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:02.406 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:02.406 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:02.415 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:02.451 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:02.457+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:02.517 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:02.517 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:02.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:02.517 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:02.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.517 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:02.518 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:02.518 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.896262453000006, 2024-04-24T14:34:02.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:34:02.518 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.518 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:02.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:02.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:02.518 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:34:02.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.518 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:02.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:34:02.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:34:02.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:02.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:34:02.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:34:02.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:02.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:34:02.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:34:02.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.896085365999994, 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.523 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:34:02.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:34:02.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:34:02.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:34:02.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:34:02.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:34:02.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:02.528 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.895981452000001, 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:02.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:34:02.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:02.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:34:02.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:34:02.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:34:02.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:34:02.533 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.895593000999995, 2024-04-24T14:34:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:34:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:02.534 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:34:02.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:02.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:34:02.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:34:02.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:34:02.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:34:02.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:02.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:34:02.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:34:02.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:02.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:34:02.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:34:02.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:02.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:34:02.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:34:02.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:02.539 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.895507789000007, 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:34:02.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:34:02.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:34:02.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:34:02.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:34:02.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:34:02.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.544 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.895270908000001, 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:02.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:02.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:34:02.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:02.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:34:02.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:34:02.550 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.550 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.550 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.550 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:02.550 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:02.550 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.895189385999998, 2024-04-24T14:34:02.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:34:02.550 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:34:02.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:34:02.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:02.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:34:02.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:34:02.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:02.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:34:02.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:34:02.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.555 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.556 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:02.556 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:02.556 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.463277039000005, 2024-04-24T14:34:02.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:34:02.556 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.556 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:02.556 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.556 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:02.556 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:02.556 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:34:02.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.556 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:02.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:02.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:34:02.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:02.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:34:02.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:34:02.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:34:02.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:34:02.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:34:02.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:34:02.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:34:02.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:34:02.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:34:02.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:34:02.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:02.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:34:02.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:34:02.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:34:02.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:34:02.561 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.561 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.561 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.562 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:02.562 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:02.562 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.463248738999994, 2024-04-24T14:34:02.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:34:02.562 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.562 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:02.562 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.562 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:02.562 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:02.562 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:34:02.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.562 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:02.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:02.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:34:02.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:02.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:34:02.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:34:02.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:34:02.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:34:02.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:34:02.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:34:02.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:34:02.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:34:02.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:34:02.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:34:02.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:02.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:34:02.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:34:02.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:02.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:34:02.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:34:02.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:02.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:34:02.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:34:02.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:02.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:34:02.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:34:02.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:34:02.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:34:02.567 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.567 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.567 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.567 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:02.567 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:02.568 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.463161034999999, 2024-04-24T14:34:02.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:34:02.568 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.568 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:02.568 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.568 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:02.568 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:02.568 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:34:02.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.568 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:02.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:02.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:34:02.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:02.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:34:02.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:34:02.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:34:02.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:34:02.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:34:02.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:34:02.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:34:02.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:34:02.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:34:02.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:34:02.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:02.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:34:02.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:34:02.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:02.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:34:02.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:34:02.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:02.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:34:02.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:34:02.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:02.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:34:02.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:34:02.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:34:02.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:34:02.573 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.573 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.573 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.573 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:02.573 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.463038233999995, 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:02.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:34:02.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:02.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:34:02.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:34:02.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:34:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:34:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:34:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:34:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:34:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:02.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:34:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:34:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:34:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:34:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:34:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:34:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:34:02.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:34:02.579 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.579 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.579 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.579 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:34:02.579 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:02.579 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 30.156793673999999, 2024-04-24T14:34:02.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:34:02.579 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.579 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:02.579 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:34:02.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:34:02.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:34:02.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:34:02.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:34:02.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:34:02.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:34:02.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:34:02.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:34:02.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:34:02.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:02.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:34:02.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:34:02.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.462669675000001, 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:34:02.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:02.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:34:02.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:34:02.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:34:02.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:34:02.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:34:02.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:34:02.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:34:02.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:34:02.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:34:02.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:34:02.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.589 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.590 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:02.590 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:02.590 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.5033097089999998, 2024-04-24T14:34:02.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:34:02.590 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.590 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.590 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.590 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:02.590 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:02.590 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:34:02.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.590 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:34:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:02.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:34:02.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:34:02.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:34:02.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:34:02.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:34:02.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:34:02.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:02.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:34:02.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:34:02.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:34:02.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:34:02.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:34:02.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:34:02.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:34:02.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:34:02.595 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.595 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.595 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.595 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:02.595 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:02.595 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.503229154, 2024-04-24T14:34:02.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:34:02.596 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.596 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.596 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.596 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:02.596 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:02.596 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:34:02.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.596 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:02.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:02.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:34:02.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:02.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:34:02.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:34:02.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:34:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:34:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:34:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:34:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:34:02.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:34:02.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:02.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:34:02.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:34:02.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:34:02.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:34:02.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.5031814250000002, 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:02.601 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:02.602 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:34:02.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.602 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:02.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:02.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:34:02.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:34:02.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:34:02.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:34:02.606 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.606 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.606 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.606 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.5031536619999999, 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:02.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:34:02.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:34:02.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:34:02.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:02.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:34:02.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:34:02.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:34:02.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:34:02.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:02.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.5031142270000002, 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:02.612 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:34:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:34:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:34:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:34:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:34:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:34:02.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:34:02.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:34:02.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:34:02.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:02.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:34:02.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:34:02.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:34:02.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.617 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:02.618 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:02.618 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.5030892429999998, 2024-04-24T14:34:02.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:34:02.618 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.618 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.618 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.618 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:02.618 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:02.618 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:34:02.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.618 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:34:02.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:34:02.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:34:02.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:34:02.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:34:02.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:34:02.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:02.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:34:02.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:34:02.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:34:02.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:34:02.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:34:02.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:34:02.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:34:02.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:34:02.623 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.623 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.623 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.623 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:02.623 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:02.623 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.503063584, 2024-04-24T14:34:02.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:34:02.623 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.623 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:02.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:34:02.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:34:02.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:34:02.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:02.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:34:02.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:34:02.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:02.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.628 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.629 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:02.630 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:02.630 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:02.635+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:02.731 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:02.735+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:02.735 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:02.735 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:02.735 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:02.735 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:02.744 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:02.832 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:02.837+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:02.843 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:02.843 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:02.843 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:02.843 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:02.843 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:02.852 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:02.877 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-24T14:34:02.921 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:34:02.922 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^18395\b' 2024-04-24T14:34:02.922 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^18395\b' 2024-04-24T14:34:02.934 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:02.939+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:02.960 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:379 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.299807744999999, 2024-04-24T14:34:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068078410000000002, 2024-04-24T14:34:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:02.961 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:02.962 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 379 2024-04-24T14:34:02.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.962 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:02.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:02.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664240+0000", 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6539999999999999e-06 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664245+0000", 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2179999999999998e-06 2024-04-24T14:34:02.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664249+0000", 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2279999999999998e-06 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670245+0000", 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059962430000000001 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670400+0000", 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015535399999999999 2024-04-24T14:34:02.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671046+0000", 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064591599999999996 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:380 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.299761117999999, 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076160830000000001, 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.965 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.966 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.966 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:02.966 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:02.966 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 380 2024-04-24T14:34:02.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.966 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:34:02.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:34:02.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664284+0000", 2024-04-24T14:34:02.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:02.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664287+0000", 2024-04-24T14:34:02.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9739999999999998e-06 2024-04-24T14:34:02.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664302+0000", 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5214999999999999e-05 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664307+0000", 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6859999999999997e-06 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671098+0000", 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067912479999999997 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671253+0000", 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015427400000000001 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671901+0000", 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064845600000000005 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:02.969 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.299731572999999, 2024-04-24T14:34:02.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:34:02.970 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.970 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.970 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.970 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:02.970 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:02.970 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:34:02.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.970 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:02.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:34:02.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:34:02.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:34:02.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:34:02.973 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.973 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.973 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.974 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:382 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:02.974 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:34:02.974 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.296730090000001, 2024-04-24T14:34:02.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067029910000000002, 2024-04-24T14:34:02.974 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.974 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.974 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.974 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:02.974 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:02.974 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 382 2024-04-24T14:34:02.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.974 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667315+0000", 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667321+0000", 2024-04-24T14:34:02.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2099999999999998e-06 2024-04-24T14:34:02.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667336+0000", 2024-04-24T14:34:02.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4612e-05 2024-04-24T14:34:02.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667343+0000", 2024-04-24T14:34:02.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0119999999999999e-06 2024-04-24T14:34:02.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672957+0000", 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056139700000000002 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.673127+0000", 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017044599999999999 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674019+0000", 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00089165499999999999 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.296697046999999, 2024-04-24T14:34:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:34:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:34:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.979 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:02.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:02.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:02.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:02.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:34:02.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:34:02.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:34:02.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:34:02.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:34:02.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:34:02.982 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.982 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.982 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.982 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.982 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:02.982 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:02.982 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.296672861000001, 2024-04-24T14:34:02.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:34:02.982 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.982 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.982 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.982 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:02.983 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:02.983 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:34:02.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.983 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:02.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:02.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:34:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:34:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:34:02.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:34:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:34:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:34:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:34:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:34:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:34:02.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:34:02.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:34:02.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:34:02.986 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.986 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.986 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.986 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.296655477000002, 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:02.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:34:02.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:02.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:34:02.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:34:02.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:34:02.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:34:02.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:34:02.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:34:02.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:34:02.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:34:02.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:34:02.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:34:02.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:34:02.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:34:02.990 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.990 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:399 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.282245342, 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071317810000000002, 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 399 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.991 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:34:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:34:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681800+0000", 2024-04-24T14:34:02.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681804+0000", 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.715e-06 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681816+0000", 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1851000000000001e-05 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681820+0000", 2024-04-24T14:34:02.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6450000000000004e-06 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688059+0000", 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062390090000000002 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688196+0000", 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013721999999999999 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688933+0000", 2024-04-24T14:34:02.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073605000000000003 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.282219385000001, 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.995 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:02.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:02.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:02.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:02.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:02.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:02.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:02.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:34:02.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:34:02.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:34:02.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:02.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:34:02.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:34:02.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:02.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:34:02.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:34:02.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.282187086, 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:02.999 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:34:03.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:03.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:34:03.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:34:03.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:34:03.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:34:03.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:414 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.267991676000001, 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069877910000000001, 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.003 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 414 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696053+0000", 2024-04-24T14:34:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696058+0000", 2024-04-24T14:34:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7509999999999999e-06 2024-04-24T14:34:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696074+0000", 2024-04-24T14:34:03.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5743000000000002e-05 2024-04-24T14:34:03.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696093+0000", 2024-04-24T14:34:03.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9508e-05 2024-04-24T14:34:03.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702314+0000", 2024-04-24T14:34:03.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062207010000000004 2024-04-24T14:34:03.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702431+0000", 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011723299999999999 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703042+0000", 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000610752 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.267934809, 2024-04-24T14:34:03.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:03.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.009 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:34:03.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:03.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.009 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:34:03.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:34:03.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:34:03.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:34:03.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:34:03.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:34:03.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:34:03.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:34:03.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:03.011 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.267905709000001, 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:34:03.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:34:03.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:34:03.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:34:03.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:34:03.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:34:03.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.015 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.267879779000001, 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:03.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:34:03.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:34:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:34:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:34:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:34:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:34:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.020 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:429 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3538944~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:03.020 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:34:03.020 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.253598339, 2024-04-24T14:34:03.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071866789999999996, 2024-04-24T14:34:03.020 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.020 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:03.020 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:03.020 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:03.020 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:03.020 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 429 2024-04-24T14:34:03.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.020 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710447+0000", 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:03.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710451+0000", 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5930000000000002e-06 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710465+0000", 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.411e-05 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710470+0000", 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3129999999999998e-06 2024-04-24T14:34:03.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716696+0000", 2024-04-24T14:34:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006225983 2024-04-24T14:34:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716901+0000", 2024-04-24T14:34:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020550200000000001 2024-04-24T14:34:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717634+0000", 2024-04-24T14:34:03.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073292099999999998 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:430 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.252234031, 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068326970000000004, 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 430 2024-04-24T14:34:03.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711810+0000", 2024-04-24T14:34:03.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:03.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711822+0000", 2024-04-24T14:34:03.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2166e-05 2024-04-24T14:34:03.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711856+0000", 2024-04-24T14:34:03.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3816000000000003e-05 2024-04-24T14:34:03.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711873+0000", 2024-04-24T14:34:03.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6742999999999999e-05 2024-04-24T14:34:03.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717693+0000", 2024-04-24T14:34:03.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058204579999999997 2024-04-24T14:34:03.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717864+0000", 2024-04-24T14:34:03.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017128299999999999 2024-04-24T14:34:03.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718645+0000", 2024-04-24T14:34:03.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078033699999999998 2024-04-24T14:34:03.028 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.028 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.028 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.028 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:03.028 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:03.028 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.252159315, 2024-04-24T14:34:03.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:34:03.028 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.028 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:03.028 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:34:03.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:03.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:34:03.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:34:03.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.252127383000001, 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:34:03.032 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:03.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:34:03.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:03.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:34:03.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:34:03.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.252059867, 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:03.036 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:34:03.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:03.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:34:03.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:34:03.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:34:03.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:34:03.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.194646983999998, 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:34:03.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:34:03.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.042 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:34:03.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:34:03.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.042 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:34:03.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:34:03.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.043 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.044 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.044 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:03.045 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:03.045 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:34:03.045 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *18395\b' 2024-04-24T14:34:03.046 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *18395\b' 2024-04-24T14:34:03.046 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:03.046 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:34:03.046 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:03.046 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:03.047+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:03.146 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:03.149+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:03.158 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:03.158 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:03.159 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:03.159 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:03.167 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:03.247 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:03.253+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:03.312 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:03.312 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:03.312 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:03.312 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:03.312 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:03.321 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:03.348 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:03.355+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:03.421 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:34:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.353543858999998, 2024-04-24T14:34:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:34:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:34:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:34:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:34:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:34:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:34:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:34:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:34:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:34:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:34:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:34:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:34:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:34:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:34:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:34:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:34:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.3587292800000004, 2024-04-24T14:34:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:34:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:34:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:34:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:34:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:34:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:34:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:34:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:34:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:34:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:34:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:34:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:34:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:34:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.3478000219999995, 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:34:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:34:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:34:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:34:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:34:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:34:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:34:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:34:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:34:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:34:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.3368439409999997, 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:34:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:34:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:34:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:34:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:34:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.325274576, 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:34:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:34:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:109 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'60, mlcod=62'58)", 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.0821624110000005, 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036290020000000001, 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:34:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343311+0000", 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343841+0000", 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053003399999999997 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343876+0000", 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4479e-05 2024-04-24T14:34:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343888+0000", 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1623e-05 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343996+0000", 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108028 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.344044+0000", 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8081999999999998e-05 2024-04-24T14:34:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.346872+0000", 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028280789999999998 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.346942+0000", 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0405000000000001e-05 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.0721549170000007, 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:34:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:34:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:34:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:34:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:34:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.0614097489999992, 2024-04-24T14:34:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:34:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:34:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:34:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:34:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:34:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:34:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:34:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.0490434729999993, 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:34:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:34:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:34:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:34:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:34:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:34:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.924107352, 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:34:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:34:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:138 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'70, mlcod=62'68)", 2024-04-24T14:34:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:34:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.9137211979999993, 2024-04-24T14:34:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036613330000000001, 2024-04-24T14:34:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:34:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511753+0000", 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512370+0000", 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061776500000000005 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512395+0000", 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4549000000000002e-05 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512404+0000", 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1130000000000007e-06 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512517+0000", 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112679 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512581+0000", 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4274000000000002e-05 2024-04-24T14:34:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.515303+0000", 2024-04-24T14:34:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027220640000000002 2024-04-24T14:34:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.515416+0000", 2024-04-24T14:34:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112644 2024-04-24T14:34:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.9047195089999995, 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:34:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:34:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:34:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:34:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:34:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:34:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:34:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:34:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:34:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:34:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:34:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:34:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.8949916659999992, 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:34:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:34:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:34:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:34:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:34:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:34:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:34:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:34:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:34:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:172 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'76, mlcod=62'74)", 2024-04-24T14:34:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:34:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.7447243120000007, 2024-04-24T14:34:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032080400000000001, 2024-04-24T14:34:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680750+0000", 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681277+0000", 2024-04-24T14:34:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052776400000000001 2024-04-24T14:34:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681304+0000", 2024-04-24T14:34:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6948e-05 2024-04-24T14:34:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681315+0000", 2024-04-24T14:34:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0648e-05 2024-04-24T14:34:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681413+0000", 2024-04-24T14:34:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7926999999999999e-05 2024-04-24T14:34:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681470+0000", 2024-04-24T14:34:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7401999999999997e-05 2024-04-24T14:34:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.683923+0000", 2024-04-24T14:34:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024526790000000001 2024-04-24T14:34:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.683959+0000", 2024-04-24T14:34:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6327e-05 2024-04-24T14:34:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:34:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.7348819859999995, 2024-04-24T14:34:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:34:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:34:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:34:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:34:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:34:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:34:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.7251529899999998, 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:34:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:34:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:34:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:34:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:34:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:34:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:178 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'82, mlcod=62'80)", 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.7146856140000004, 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029527640000000001, 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:34:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710789+0000", 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711538+0000", 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00074893099999999999 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711565+0000", 2024-04-24T14:34:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7526999999999999e-05 2024-04-24T14:34:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711576+0000", 2024-04-24T14:34:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0553999999999999e-05 2024-04-24T14:34:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711633+0000", 2024-04-24T14:34:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7258e-05 2024-04-24T14:34:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711653+0000", 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0460999999999999e-05 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.713712+0000", 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020587790000000002 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.713743+0000", 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0519999999999999e-05 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 4.3901821549999998, 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:34:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:34:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.54603140699999997, 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:34:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:34:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.957086+0000", 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 0.468390062, 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029645230000000002, 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957086+0000", 2024-04-24T14:34:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957086+0000", 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957084+0000", 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957106+0000", 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1866e-05 2024-04-24T14:34:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957130+0000", 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4266000000000001e-05 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957140+0000", 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.007e-05 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957264+0000", 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012356699999999999 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.960050+0000", 2024-04-24T14:34:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027863570000000002 2024-04-24T14:34:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:03.502 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:03.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:03.504 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd16809 --snap snap 2024-04-24T14:34:03.505 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:34:03.505 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:34:03.505 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:34:03.505 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:34:03.505 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:34:03.505 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:34:03.505 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd16809 --snap snap 2024-04-24T14:34:03.505 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:03.507+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:03.604 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:03.607+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:03.612 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:03.612 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:03.612 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:03.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:03.649 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:34:03.649 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd ns 2024-04-24T14:34:03.649 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:03.649 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns=ns 2024-04-24T14:34:03.650 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:03.650 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:03.706 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:03.709+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:03.721 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=18459 2024-04-24T14:34:03.722 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 18459 2024-04-24T14:34:03.722 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 18459 -C rbd-nbd 2024-04-24T14:34:03.724 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:03.724 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:03.724 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:03.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:03.733 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:03.733 INFO:tasks.workunit.client.0.smithi173.stdout: 18459 ? 00:00:00 rbd-nbd 2024-04-24T14:34:03.734 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd16809 --snap snap' 18459 2024-04-24T14:34:03.734 INFO:tasks.workunit.client.0.smithi173.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd16809 --snap snap' 2024-04-24T14:34:03.734 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=18459 2024-04-24T14:34:03.734 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd16809 --snap snap 2024-04-24T14:34:03.734 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:03.734 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:03.735 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:03.735 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:03.736 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:03.736 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:03.736 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd16809 --snap snap 2024-04-24T14:34:03.807 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:03.811+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:03.829 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:34:03.829 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^18459\b' 2024-04-24T14:34:03.830 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^18459\b' 2024-04-24T14:34:03.842 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.9670343080000001, 2024-04-24T14:34:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:34:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:34:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:34:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:34:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:34:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:34:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:34:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:34:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:34:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:34:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:34:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:34:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:34:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:34:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.967005323, 2024-04-24T14:34:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:34:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:34:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:34:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:34:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:34:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:34:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:34:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:34:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:34:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:34:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:34:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:34:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.966986838, 2024-04-24T14:34:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:34:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:34:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:34:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:34:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:34:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:34:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:34:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:34:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:34:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:34:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:34:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:34:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:34:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:34:03.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.9669715569999999, 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:34:03.859 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:03.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:34:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:34:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:34:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:34:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:34:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:34:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:34:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:34:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:34:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:34:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:34:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:34:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:34:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:34:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.864 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.966955776, 2024-04-24T14:34:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:34:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:34:03.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:34:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:34:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:34:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:34:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:34:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:34:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:34:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:34:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:34:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:34:03.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:34:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:34:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:34:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:34:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.966941488, 2024-04-24T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:03.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:34:03.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.966921736, 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.876 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:03.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:34:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:34:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:34:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:34:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:34:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:34:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:34:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:34:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:34:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:34:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:34:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:34:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.9669042889999999, 2024-04-24T14:34:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:34:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:34:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:34:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:34:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:34:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:34:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:34:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:34:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:34:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:34:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:34:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:34:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:34:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:34:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:34:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:34:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4d9853ae:::rbd_id.INVALIDIMAGE:head, version: 53'12, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4d9853ae:::rbd_id.INVALIDIMAGE:head@53'12, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.94162575800000003, 2024-04-24T14:34:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075925020000000001, 2024-04-24T14:34:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905243+0000", 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905253+0000", 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1619999999999997e-06 2024-04-24T14:34:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905262+0000", 2024-04-24T14:34:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2399999999999996e-06 2024-04-24T14:34:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905268+0000", 2024-04-24T14:34:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2530000000000001e-06 2024-04-24T14:34:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912457+0000", 2024-04-24T14:34:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071890779999999998 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912462+0000", 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5159999999999996e-06 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912836+0000", 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037484699999999998 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 62'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@62'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.91800682899999997, 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036829689999999999, 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928862+0000", 2024-04-24T14:34:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930264+0000", 2024-04-24T14:34:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014016059999999999 2024-04-24T14:34:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930281+0000", 2024-04-24T14:34:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7859999999999998e-05 2024-04-24T14:34:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930291+0000", 2024-04-24T14:34:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9299999999999998e-06 2024-04-24T14:34:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930894+0000", 2024-04-24T14:34:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060250899999999999 2024-04-24T14:34:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930903+0000", 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4599999999999992e-06 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.932546+0000", 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016425019999999999 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head, version: 62'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head@62'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.91229188100000003, 2024-04-24T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037783690000000002, 2024-04-24T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934576+0000", 2024-04-24T14:34:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936033+0000", 2024-04-24T14:34:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001456658 2024-04-24T14:34:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936071+0000", 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7752e-05 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936087+0000", 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5919000000000001e-05 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936226+0000", 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013910800000000001 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936240+0000", 2024-04-24T14:34:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.456e-05 2024-04-24T14:34:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938356+0000", 2024-04-24T14:34:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021157540000000001 2024-04-24T14:34:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 62'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@62'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.91124535299999998, 2024-04-24T14:34:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052678259999999998, 2024-04-24T14:34:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935623+0000", 2024-04-24T14:34:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937055+0000", 2024-04-24T14:34:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014314480000000001 2024-04-24T14:34:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937085+0000", 2024-04-24T14:34:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0204000000000001e-05 2024-04-24T14:34:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937097+0000", 2024-04-24T14:34:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1708e-05 2024-04-24T14:34:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938410+0000", 2024-04-24T14:34:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013128910000000001 2024-04-24T14:34:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938419+0000", 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1740000000000006e-06 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.940892+0000", 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024734990000000001 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 62'82, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@62'82, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.901733962, 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003877447, 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945134+0000", 2024-04-24T14:34:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946323+0000", 2024-04-24T14:34:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001188635 2024-04-24T14:34:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946352+0000", 2024-04-24T14:34:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9238e-05 2024-04-24T14:34:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946374+0000", 2024-04-24T14:34:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1396999999999999e-05 2024-04-24T14:34:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.947098+0000", 2024-04-24T14:34:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072458199999999998 2024-04-24T14:34:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.947109+0000", 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0557000000000001e-05 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.949013+0000", 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019044929999999999 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.89527697699999997, 2024-04-24T14:34:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041579369999999996, 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951585+0000", 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999928 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953339+0000", 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017540019999999999 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953363+0000", 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3178000000000001e-05 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953374+0000", 2024-04-24T14:34:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1015e-05 2024-04-24T14:34:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953522+0000", 2024-04-24T14:34:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014857800000000001 2024-04-24T14:34:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953565+0000", 2024-04-24T14:34:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2511e-05 2024-04-24T14:34:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955751+0000", 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002185958 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 62'82, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@62'82, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.894862461, 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005349749, 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952006+0000", 2024-04-24T14:34:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953689+0000", 2024-04-24T14:34:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00168326 2024-04-24T14:34:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953717+0000", 2024-04-24T14:34:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7994e-05 2024-04-24T14:34:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953746+0000", 2024-04-24T14:34:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8059999999999999e-05 2024-04-24T14:34:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955798+0000", 2024-04-24T14:34:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002052089 2024-04-24T14:34:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955812+0000", 2024-04-24T14:34:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2024-04-24T14:34:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.957357+0000", 2024-04-24T14:34:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001545537 2024-04-24T14:34:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.88607234300000004, 2024-04-24T14:34:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051567999999999996, 2024-04-24T14:34:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960796+0000", 2024-04-24T14:34:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962459+0000", 2024-04-24T14:34:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016622259999999999 2024-04-24T14:34:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962481+0000", 2024-04-24T14:34:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2359999999999999e-05 2024-04-24T14:34:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962496+0000", 2024-04-24T14:34:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.523e-05 2024-04-24T14:34:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.964299+0000", 2024-04-24T14:34:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018025129999999999 2024-04-24T14:34:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.964316+0000", 2024-04-24T14:34:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7436999999999999e-05 2024-04-24T14:34:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.965954+0000", 2024-04-24T14:34:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016381099999999999 2024-04-24T14:34:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head, version: 62'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head@62'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.85306615399999997, 2024-04-24T14:34:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003994821, 2024-04-24T14:34:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993802+0000", 2024-04-24T14:34:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994846+0000", 2024-04-24T14:34:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00104368 2024-04-24T14:34:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994884+0000", 2024-04-24T14:34:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7796000000000002e-05 2024-04-24T14:34:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994900+0000", 2024-04-24T14:34:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5917e-05 2024-04-24T14:34:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995046+0000", 2024-04-24T14:34:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014653499999999999 2024-04-24T14:34:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995055+0000", 2024-04-24T14:34:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0759999999999992e-06 2024-04-24T14:34:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.997798+0000", 2024-04-24T14:34:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027432709999999998 2024-04-24T14:34:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(2.2 68/67 [PushOp(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 49'1, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@49'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.85275869599999998, 2024-04-24T14:34:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038339310000000001, 2024-04-24T14:34:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994110+0000", 2024-04-24T14:34:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994914+0000", 2024-04-24T14:34:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080383099999999997 2024-04-24T14:34:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994943+0000", 2024-04-24T14:34:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9475000000000002e-05 2024-04-24T14:34:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994955+0000", 2024-04-24T14:34:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1525000000000001e-05 2024-04-24T14:34:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995123+0000", 2024-04-24T14:34:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016812899999999999 2024-04-24T14:34:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995143+0000", 2024-04-24T14:34:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9724999999999998e-05 2024-04-24T14:34:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.997945+0000", 2024-04-24T14:34:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028023890000000002 2024-04-24T14:34:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head, version: 62'144, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head@62'144, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.81245350100000002, 2024-04-24T14:34:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041651270000000002, 2024-04-24T14:34:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034415+0000", 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035025+0000", 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060999700000000003 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035077+0000", 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2132999999999997e-05 2024-04-24T14:34:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035100+0000", 2024-04-24T14:34:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2690000000000001e-05 2024-04-24T14:34:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035293+0000", 2024-04-24T14:34:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019358900000000001 2024-04-24T14:34:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035311+0000", 2024-04-24T14:34:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7952999999999999e-05 2024-04-24T14:34:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.038581+0000", 2024-04-24T14:34:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032700670000000002 2024-04-24T14:34:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head, version: 62'144, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head@62'144, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.79828806600000002, 2024-04-24T14:34:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039835970000000002, 2024-04-24T14:34:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048581+0000", 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049792+0000", 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012111579999999999 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049823+0000", 2024-04-24T14:34:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1137999999999999e-05 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049838+0000", 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5061e-05 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.050005+0000", 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016745400000000001 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.050022+0000", 2024-04-24T14:34:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7022000000000001e-05 2024-04-24T14:34:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.052565+0000", 2024-04-24T14:34:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025428669999999999 2024-04-24T14:34:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:03.935 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:03.935 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:03.937 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:03.937 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:34:03.937 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *18459\b' 2024-04-24T14:34:03.937 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *18459\b' 2024-04-24T14:34:03.937 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:03.937 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:34:03.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:03.938 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:03.941+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:04.038 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:04.043+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:04.088 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:04.088 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:04.089 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:04.089 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:04.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:04.142 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:04.147+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:04.195 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:04.195 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:04.195 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:04.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:04.240 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:04.243+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:04.289 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:34:04.308 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:04.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:97 3.1 3:990e5750:::rbd_header.12ff64619e2d:head [watch reconnect cookie 140276853203408 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.957467479000002, 2024-04-24T14:34:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012161632, 2024-04-24T14:34:04.309 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 97 2024-04-24T14:34:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352830+0000", 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352834+0000", 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2699999999999998e-06 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352844+0000", 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0406e-05 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352850+0000", 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6439999999999997e-06 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362391+0000", 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095408969999999996 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362683+0000", 2024-04-24T14:34:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029238000000000002 2024-04-24T14:34:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.364992+0000", 2024-04-24T14:34:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023092020000000002 2024-04-24T14:34:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.313 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:225 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.805025366999999, 2024-04-24T14:34:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010959015000000001, 2024-04-24T14:34:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 225 2024-04-24T14:34:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:04.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505275+0000", 2024-04-24T14:34:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0269999999999998e-06 2024-04-24T14:34:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505279+0000", 2024-04-24T14:34:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4279999999999998e-06 2024-04-24T14:34:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505282+0000", 2024-04-24T14:34:04.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.982e-06 2024-04-24T14:34:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515209+0000", 2024-04-24T14:34:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099266419999999994 2024-04-24T14:34:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515466+0000", 2024-04-24T14:34:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025719700000000002 2024-04-24T14:34:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.516232+0000", 2024-04-24T14:34:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076608899999999996 2024-04-24T14:34:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.553993625, 2024-04-24T14:34:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:34:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:34:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:34:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:34:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:34:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:34:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:34:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:34:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:34:04.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:34:04.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:04.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:34:04.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:34:04.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:34:04.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:34:04.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.553942791000001, 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.324 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:04.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:04.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:04.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:34:04.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:34:04.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.329 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.330 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.330 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:04.330 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:04.330 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.553906483999999, 2024-04-24T14:34:04.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:34:04.330 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.330 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.330 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.330 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.330 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.330 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:34:04.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:34:04.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:34:04.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:34:04.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:04.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:34:04.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:34:04.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:34:04.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:34:04.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:34:04.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:34:04.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:34:04.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:34:04.335 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.335 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.335 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.335 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:04.335 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.550495234, 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:34:04.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:34:04.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:34:04.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:34:04.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:34:04.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:34:04.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:04.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:34:04.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:34:04.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:34:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:34:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:34:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:34:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:34:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:34:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:34:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:34:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:34:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:34:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.550433557000002, 2024-04-24T14:34:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:34:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:34:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:34:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:34:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:34:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:34:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:34:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:34:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:34:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:34:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:34:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:34:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:34:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:34:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:34:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:34:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.550398981000001, 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:34:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:34:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:34:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:34:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:34:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:34:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:34:04.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:34:04.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:04.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:34:04.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:34:04.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:34:04.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:34:04.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:34:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:34:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:34:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:34:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:34:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:34:04.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:34:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:34:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.550369642, 2024-04-24T14:34:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:34:04.353 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.354 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.354 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.354 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.354 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.354 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:34:04.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.354 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:34:04.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:34:04.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:34:04.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:04.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:34:04.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:34:04.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:34:04.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:34:04.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:04.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:34:04.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:34:04.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:34:04.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:34:04.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:34:04.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:34:04.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:34:04.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:34:04.359 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.359 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.550293060000001, 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:04.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:04.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:34:04.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:04.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:34:04.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:34:04.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:34:04.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:34:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:34:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:34:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:34:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:34:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:04.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:34:04.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:34:04.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:34:04.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:34:04.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:04.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:34:04.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:34:04.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:34:04.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:34:04.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:34:04.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:34:04.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:34:04.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:34:04.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.366 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.366 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:04.366 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:04.366 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.550245515, 2024-04-24T14:34:04.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:34:04.366 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:04.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:34:04.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:04.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:34:04.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:34:04.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:34:04.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:34:04.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:34:04.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:34:04.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:34:04.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:34:04.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:34:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:34:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:34:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:34:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:34:04.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:34:04.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.550204872999998, 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.372 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:34:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:34:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:34:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:34:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:34:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:34:04.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:34:04.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:04.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:34:04.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:34:04.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:34:04.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:34:04.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:34:04.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:34:04.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:34:04.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:34:04.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.377 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.378 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:04.378 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:04.378 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.550187762, 2024-04-24T14:34:04.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:34:04.378 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.378 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.378 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.378 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.378 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.378 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:34:04.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.378 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:04.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:04.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:34:04.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:04.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:34:04.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:34:04.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:34:04.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:34:04.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:34:04.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:34:04.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:04.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:34:04.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:34:04.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:34:04.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:34:04.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:34:04.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:34:04.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:34:04.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:34:04.383 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.383 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.383 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.384 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:481 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:04.384 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:04.384 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.516316055000001, 2024-04-24T14:34:04.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010661718000000001, 2024-04-24T14:34:04.384 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.384 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.384 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 481 2024-04-24T14:34:04.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.384 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:04.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:04.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793981+0000", 2024-04-24T14:34:04.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:04.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793985+0000", 2024-04-24T14:34:04.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8469999999999998e-06 2024-04-24T14:34:04.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:04.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0220000000000004e-06 2024-04-24T14:34:04.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793998+0000", 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8559999999999998e-06 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803807+0000", 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098088629999999993 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803916+0000", 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010854899999999999 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.804644+0000", 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072835399999999996 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:497 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.498375743, 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013996471, 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.388 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:04.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:04.389 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 497 2024-04-24T14:34:04.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.389 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:04.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811922+0000", 2024-04-24T14:34:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811925+0000", 2024-04-24T14:34:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4120000000000001e-06 2024-04-24T14:34:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811936+0000", 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1077000000000001e-05 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811941+0000", 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7539999999999997e-06 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822289+0000", 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010347635000000001 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822457+0000", 2024-04-24T14:34:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000168257 2024-04-24T14:34:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.825919+0000", 2024-04-24T14:34:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003461979 2024-04-24T14:34:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:25 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.342817802000001, 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013349789000000001, 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 25 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967479+0000", 2024-04-24T14:34:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967669+0000", 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000189829 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967687+0000", 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8199999999999999e-05 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967695+0000", 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4869999999999996e-06 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.972501+0000", 2024-04-24T14:34:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048057509999999996 2024-04-24T14:34:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.972824+0000", 2024-04-24T14:34:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032322299999999998 2024-04-24T14:34:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.973071+0000", 2024-04-24T14:34:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024751299999999999 2024-04-24T14:34:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977624+0000", 2024-04-24T14:34:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045527570000000002 2024-04-24T14:34:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980627+0000", 2024-04-24T14:34:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003002972 2024-04-24T14:34:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980743+0000", 2024-04-24T14:34:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2024-04-24T14:34:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980830+0000", 2024-04-24T14:34:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7534999999999995e-05 2024-04-24T14:34:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:26 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.342594067, 2024-04-24T14:34:04.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013403597999999999, 2024-04-24T14:34:04.399 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.399 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:04.399 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.399 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:04.399 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:04.399 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 26 2024-04-24T14:34:04.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.399 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:04.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:04.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967703+0000", 2024-04-24T14:34:04.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:04.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967874+0000", 2024-04-24T14:34:04.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017088499999999999 2024-04-24T14:34:04.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967886+0000", 2024-04-24T14:34:04.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1744e-05 2024-04-24T14:34:04.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967893+0000", 2024-04-24T14:34:04.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2024-04-24T14:34:04.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974028+0000", 2024-04-24T14:34:04.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061356650000000002 2024-04-24T14:34:04.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974347+0000", 2024-04-24T14:34:04.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031897400000000003 2024-04-24T14:34:04.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:04.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974774+0000", 2024-04-24T14:34:04.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042661700000000001 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980930+0000", 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061563360000000001 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980945+0000", 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4864e-05 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981035+0000", 2024-04-24T14:34:04.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9390999999999998e-05 2024-04-24T14:34:04.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981108+0000", 2024-04-24T14:34:04.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3255000000000003e-05 2024-04-24T14:34:04.404 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.404 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.404 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.404 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:27 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:04.404 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:04.405 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.342396921000001, 2024-04-24T14:34:04.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013394262000000001, 2024-04-24T14:34:04.405 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.405 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:04.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:04.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:04.405 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 27 2024-04-24T14:34:04.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.405 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:04.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:04.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967900+0000", 2024-04-24T14:34:04.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:04.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968061+0000", 2024-04-24T14:34:04.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016101900000000001 2024-04-24T14:34:04.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968072+0000", 2024-04-24T14:34:04.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1012999999999999e-05 2024-04-24T14:34:04.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968079+0000", 2024-04-24T14:34:04.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.46e-06 2024-04-24T14:34:04.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.976820+0000", 2024-04-24T14:34:04.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087406289999999998 2024-04-24T14:34:04.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.976951+0000", 2024-04-24T14:34:04.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131187 2024-04-24T14:34:04.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:04.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977129+0000", 2024-04-24T14:34:04.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000178135 2024-04-24T14:34:04.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:04.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981126+0000", 2024-04-24T14:34:04.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00399709 2024-04-24T14:34:04.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:04.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981193+0000", 2024-04-24T14:34:04.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7083999999999997e-05 2024-04-24T14:34:04.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:04.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981251+0000", 2024-04-24T14:34:04.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7754000000000003e-05 2024-04-24T14:34:04.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981296+0000", 2024-04-24T14:34:04.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4765000000000003e-05 2024-04-24T14:34:04.411 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.411 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.411 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.411 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:28 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:04.412 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:04.412 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.342211688999999, 2024-04-24T14:34:04.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013427247999999999, 2024-04-24T14:34:04.412 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.412 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:04.412 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.412 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:04.412 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:04.412 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 28 2024-04-24T14:34:04.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.413 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:04.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:04.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968086+0000", 2024-04-24T14:34:04.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:04.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968247+0000", 2024-04-24T14:34:04.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016121199999999999 2024-04-24T14:34:04.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968260+0000", 2024-04-24T14:34:04.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.275e-05 2024-04-24T14:34:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968266+0000", 2024-04-24T14:34:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3770000000000001e-06 2024-04-24T14:34:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977699+0000", 2024-04-24T14:34:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094331439999999992 2024-04-24T14:34:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977894+0000", 2024-04-24T14:34:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194945 2024-04-24T14:34:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.978258+0000", 2024-04-24T14:34:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036359400000000002 2024-04-24T14:34:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981140+0000", 2024-04-24T14:34:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028821540000000001 2024-04-24T14:34:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981360+0000", 2024-04-24T14:34:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022056300000000001 2024-04-24T14:34:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981444+0000", 2024-04-24T14:34:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3733999999999995e-05 2024-04-24T14:34:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981514+0000", 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9702000000000005e-05 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:29 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.342024844000001, 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014492283, 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 29 2024-04-24T14:34:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968434+0000", 2024-04-24T14:34:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016130199999999999 2024-04-24T14:34:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968484+0000", 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0003000000000002e-05 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968499+0000", 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4628e-05 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979299+0000", 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010800624999999999 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979488+0000", 2024-04-24T14:34:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000189037 2024-04-24T14:34:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979784+0000", 2024-04-24T14:34:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029562000000000002 2024-04-24T14:34:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982305+0000", 2024-04-24T14:34:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025212339999999998 2024-04-24T14:34:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982680+0000", 2024-04-24T14:34:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000374762 2024-04-24T14:34:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982731+0000", 2024-04-24T14:34:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1146999999999997e-05 2024-04-24T14:34:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982766+0000", 2024-04-24T14:34:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4755000000000002e-05 2024-04-24T14:34:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:04.426 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:04.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:04.429 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:04.429 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:34:04.429 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:34:04.429 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:34:04.429 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:34:04.429 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:34:04.429 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:34:04.429 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:34:04.430 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:34:04.430 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:34:04.430 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:04.433+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:04.446 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:04.474 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:04.500 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:04.522 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:04.528 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:04.533+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:04.544 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:34:04.544 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:34:04.545 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:04.545 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:04.545 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:04.545 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:04.546 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:04.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:04.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:04.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:04.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:04.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:04.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:04.550 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:04.551 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:04.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:04.610 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=18523 2024-04-24T14:34:04.610 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 18523 2024-04-24T14:34:04.610 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 18523 -C rbd-nbd 2024-04-24T14:34:04.615 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:04.615 INFO:tasks.workunit.client.0.smithi173.stdout: 18523 ? 00:00:00 rbd-nbd 2024-04-24T14:34:04.616 INFO:tasks.workunit.client.0.smithi173.stderr:+ pid=18523 2024-04-24T14:34:04.616 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd16809 2024-04-24T14:34:04.616 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:34:04.617 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:34:04.619 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:34:04.619 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:34:04.620 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:34:04.620 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:34:04.620 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd16809 2024-04-24T14:34:04.628 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:04.633+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:04.704 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:04.704 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:04.713 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:04.729 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:04.731+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:04.811 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:04.812 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:04.822 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:04.857 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd1 2024-04-24T14:34:04.857 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd ns 2024-04-24T14:34:04.857 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:04.858 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns=ns 2024-04-24T14:34:04.859 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:04.859 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd1'\'']/id' 2024-04-24T14:34:04.867 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:04.873+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:04.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T14:34:04.930 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:04.930 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:04.930 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:04.933+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:04.940 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:04.941 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=18571 2024-04-24T14:34:04.941 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 18571 2024-04-24T14:34:04.941 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 18571 -C rbd-nbd 2024-04-24T14:34:04.946 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: 18523 ? 00:00:00 rbd-nbd 2024-04-24T14:34:04.947 INFO:tasks.workunit.client.0.smithi173.stdout: 18571 ? 00:00:00 rbd-nbd 2024-04-24T14:34:04.947 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device rbd/ns/testrbdnbd16809 18571 2024-04-24T14:34:04.947 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=rbd/ns/testrbdnbd16809 2024-04-24T14:34:04.947 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=18571 2024-04-24T14:34:04.948 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd16809 2024-04-24T14:34:04.948 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:04.948 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:04.949 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:04.949 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:04.950 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:04.950 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:04.951 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd16809 2024-04-24T14:34:05.031 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:05.035+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:05.033 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:34:05.033 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^18571\b' 2024-04-24T14:34:05.033 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^18571\b' 2024-04-24T14:34:05.043 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:05.045 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:05.053 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:05.090 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:05.090 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:34:05.091 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *18571\b' 2024-04-24T14:34:05.091 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *18571\b' 2024-04-24T14:34:05.098 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:05.098 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:34:05.131 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:05.135+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:05.150 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:05.150 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:05.159 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:05.183 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [5, 4] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 1] 2024-04-24T14:34:05.184 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:34:05.184 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-04-24T14:34:05.184 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-04-24T14:34:05.233 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:05.237+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:05.260 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:05.260 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:05.269 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:05.333 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:05.337+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:05.370 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:05.370 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:05.379 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:05.434 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:05.439+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:05.479 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-04-24T14:34:05.483+0000 7ff9c25de700 -1 mon.a@0(leader).osd e70 definitely_dead 0 2024-04-24T14:34:05.527 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:05.527 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:05.534 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:05.537+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:05.536 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:05.600 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:34:05.600 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device rbd/testrbdnbd16809 18523 2024-04-24T14:34:05.600 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=rbd/testrbdnbd16809 2024-04-24T14:34:05.600 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=18523 2024-04-24T14:34:05.600 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap rbd/testrbdnbd16809 2024-04-24T14:34:05.601 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:05.601 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:05.602 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:05.603 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:05.606 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:05.606 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:05.606 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/testrbdnbd16809 2024-04-24T14:34:05.635 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:05.635 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:05.635 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:05.639+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:05.644 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:05.695 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:34:05.695 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^18523\b' 2024-04-24T14:34:05.695 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^18523\b' 2024-04-24T14:34:05.736 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:05.741+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:05.744 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:05.744 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:05.748 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:05.748 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:34:05.749 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *18523\b' 2024-04-24T14:34:05.749 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *18523\b' 2024-04-24T14:34:05.753 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:05.759 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:05.759 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:34:05.837 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:05.839+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:05.847 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-24T14:34:05.849 INFO:teuthology.orchestra.run.smithi090.stderr:marked in osd.4. 2024-04-24T14:34:05.859 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-04-24T14:34:05.900 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:05.901 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:05.909 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:05.937 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:05.939+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:06.017 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:06.017 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:06.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:06.037 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:06.041+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:06.132 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:06.132 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:06.139 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:06.143+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:06.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:06.239 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:06.243+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:06.261 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' rbd = rbd ']' 2024-04-24T14:34:06.261 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd16809 2024-04-24T14:34:06.261 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:34:06.262 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:34:06.264 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:34:06.264 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:34:06.267 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:34:06.267 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:34:06.267 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd16809 2024-04-24T14:34:06.279 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:06.280 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:06.289 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:06.341 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:06.345+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:06.384 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:06.385 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:06.394 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:06.441 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:06.443+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:06.493 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:06.493 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:06.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:06.546 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:06.551+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:06.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:34:06.603 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:06.604 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:06.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:06.643 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:06.649+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:06.690 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:06.709 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:06.727 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:06.743 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:06.749+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:06.747 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:06.747 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:06.747 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:06.747 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:06.748 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:06.756 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:06.774 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:06.797 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:06.799 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:06.799 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:06.799 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:06.799 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:06.799 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:06.799 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:06.844 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:06.849+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:06.856 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:06.856 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:06.856 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:06.856 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:06.856 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:06.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:06.945 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:06.949+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:06.962 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:192 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.495370474000001, 2024-04-24T14:34:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010036329, 2024-04-24T14:34:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192 2024-04-24T14:34:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471099+0000", 2024-04-24T14:34:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471103+0000", 2024-04-24T14:34:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8800000000000001e-06 2024-04-24T14:34:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471111+0000", 2024-04-24T14:34:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2930000000000001e-06 2024-04-24T14:34:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471116+0000", 2024-04-24T14:34:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1800000000000004e-06 2024-04-24T14:34:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480338+0000", 2024-04-24T14:34:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092213069999999998 2024-04-24T14:34:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480467+0000", 2024-04-24T14:34:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012901500000000001 2024-04-24T14:34:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481136+0000", 2024-04-24T14:34:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066948400000000003 2024-04-24T14:34:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.495347438, 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:34:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:34:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:34:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:34:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:34:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:34:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:34:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:34:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:34:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:34:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:34:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:34:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:34:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.119990734999998, 2024-04-24T14:34:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:34:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:34:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:34:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:34:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:34:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:34:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:34:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:34:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:526 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.118603212, 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010568783999999999, 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 526 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847866+0000", 2024-04-24T14:34:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847871+0000", 2024-04-24T14:34:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1590000000000001e-06 2024-04-24T14:34:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847883+0000", 2024-04-24T14:34:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.172e-05 2024-04-24T14:34:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847890+0000", 2024-04-24T14:34:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6950000000000003e-06 2024-04-24T14:34:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857418+0000", 2024-04-24T14:34:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095285909999999995 2024-04-24T14:34:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857588+0000", 2024-04-24T14:34:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169541 2024-04-24T14:34:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858436+0000", 2024-04-24T14:34:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084814700000000005 2024-04-24T14:34:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.118562580999999, 2024-04-24T14:34:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:34:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:34:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:34:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:34:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:34:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:34:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:34:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:34:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:34:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:34:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:34:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:34:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:34:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:34:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:34:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.118535491999999, 2024-04-24T14:34:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:34:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:34:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:34:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:34:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:34:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:34:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:34:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:34:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:34:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:34:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:34:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:34:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:34:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:34:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:34:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.118511524999999, 2024-04-24T14:34:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:34:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:34:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:34:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:34:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:34:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:34:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:34:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:34:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:34:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:34:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:34:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:34:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:34:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:34:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:34:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.104527546, 2024-04-24T14:34:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:34:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:34:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:34:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:34:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:34:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:34:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:34:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:34:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:34:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:34:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:34:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:34:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:34:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:34:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:34:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:34:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:34:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:34:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:34:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:34:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:34:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:34:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.104411235000001, 2024-04-24T14:34:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:34:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:34:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:34:07.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:34:07.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:34:07.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:34:07.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:34:07.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:34:07.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:34:07.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:34:07.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:34:07.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:34:07.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:34:07.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:34:07.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:34:07.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:34:07.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:07.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:34:07.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:34:07.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:07.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:34:07.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:34:07.020 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.020 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.020 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.020 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:07.020 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:07.020 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.104363577000001, 2024-04-24T14:34:07.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:34:07.020 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:07.020 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:07.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:07.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:07.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.021 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:34:07.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.021 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:07.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:07.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:07.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:07.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:07.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:07.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:34:07.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:07.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:34:07.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:34:07.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:07.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:34:07.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:34:07.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:34:07.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:34:07.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:34:07.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:34:07.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:34:07.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:34:07.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:07.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:34:07.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:34:07.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:07.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:34:07.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:34:07.025 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.025 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.026 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.026 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:07.026 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:07.026 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.100326279000001, 2024-04-24T14:34:07.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:34:07.026 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:07.026 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:07.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:07.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:07.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:34:07.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:07.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:07.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:34:07.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:34:07.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:07.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:34:07.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:34:07.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:34:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:34:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:34:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:34:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:34:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:34:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:34:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:34:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:34:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:34:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.032 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:07.032 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:07.032 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.100262520000001, 2024-04-24T14:34:07.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:34:07.032 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:07.032 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:07.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:07.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:07.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.032 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:34:07.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.032 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:07.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:34:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:34:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:34:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:34:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:34:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:34:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:34:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:34:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:34:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:34:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:34:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:34:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:34:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:34:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.100237029999999, 2024-04-24T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.038 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:07.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:07.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:07.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:34:07.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:07.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:34:07.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:34:07.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.100213279999998, 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:07.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:07.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:07.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:07.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:07.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:34:07.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:07.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:34:07.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:34:07.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:07.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:34:07.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:34:07.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:34:07.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:34:07.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:34:07.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:34:07.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:34:07.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:34:07.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:34:07.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:34:07.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:34:07.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:34:07.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:34:07.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:34:07.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.100174766999999, 2024-04-24T14:34:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:34:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:34:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:34:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:34:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:34:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:34:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:34:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:34:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:34:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:34:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:34:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:34:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:34:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:34:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:34:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:34:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:34:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.10014657, 2024-04-24T14:34:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:34:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:34:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:34:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:34:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:34:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:34:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:34:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:34:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:34:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:34:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:34:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:34:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:34:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:34:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:34:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:34:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:34:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.100125113000001, 2024-04-24T14:34:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:34:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:34:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:34:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:34:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:34:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:34:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:34:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:34:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:34:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:34:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:34:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:34:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:34:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:34:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:34:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:34:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:34:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:34:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.100106271000001, 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:34:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:34:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:34:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:34:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:34:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:34:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:34:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:34:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:34:07.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:34:07.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:34:07.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:34:07.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:34:07.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:34:07.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:34:07.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:34:07.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:34:07.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:34:07.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:07.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:34:07.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:34:07.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:07.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:34:07.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:34:07.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.074 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.074 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:561 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:07.074 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:07.075 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.058143651000002, 2024-04-24T14:34:07.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01066806, 2024-04-24T14:34:07.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:07.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:07.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:07.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:07.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.075 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 561 2024-04-24T14:34:07.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.075 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:07.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908326+0000", 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:07.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908329+0000", 2024-04-24T14:34:07.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.802e-06 2024-04-24T14:34:07.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:07.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908338+0000", 2024-04-24T14:34:07.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6260000000000001e-06 2024-04-24T14:34:07.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908345+0000", 2024-04-24T14:34:07.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1389999999999998e-06 2024-04-24T14:34:07.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.917978+0000", 2024-04-24T14:34:07.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096329840000000007 2024-04-24T14:34:07.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:07.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918178+0000", 2024-04-24T14:34:07.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019999199999999999 2024-04-24T14:34:07.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:07.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918995+0000", 2024-04-24T14:34:07.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081708099999999999 2024-04-24T14:34:07.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.079 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.079 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:07.079 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:07.079 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.887274122, 2024-04-24T14:34:07.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:34:07.079 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:07.079 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:07.079 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:07.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:07.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:34:07.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:07.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:07.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:34:07.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:34:07.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:07.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:34:07.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:07.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:34:07.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:34:07.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:34:07.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:34:07.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:07.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:34:07.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:34:07.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:07.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:34:07.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:34:07.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:07.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:07.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:07.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:34:07.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:34:07.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.085 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:07.085 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:07.085 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:07.086 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:07.087 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:07.091+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:07.188 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:07.191+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:07.195 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:07.195 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:07.195 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:07.195 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:07.208 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:07.288 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:07.293+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:07.311 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:07.311 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:07.311 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:07.311 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:07.311 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:07.320 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:07.389 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:07.393+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:07.419 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:07.419 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:07.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:07.419 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:07.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 104.796336934, 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:07.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:07.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:34:07.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:07.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:34:07.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:34:07.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:34:07.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:34:07.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:34:07.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:34:07.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:34:07.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:34:07.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:34:07.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:34:07.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:07.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:34:07.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:34:07.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:07.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:34:07.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:34:07.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:07.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:34:07.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:34:07.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:07.425 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 104.796159847, 2024-04-24T14:34:07.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:34:07.426 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.426 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:07.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:07.426 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:07.426 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:34:07.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.426 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:07.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:07.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:34:07.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:07.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:34:07.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:34:07.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:34:07.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:34:07.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:34:07.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:34:07.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:34:07.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:34:07.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:34:07.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:34:07.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:07.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:34:07.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:34:07.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:07.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:34:07.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:34:07.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:07.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:34:07.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:34:07.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:07.431 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 104.79605593300001, 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:07.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:34:07.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:07.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:34:07.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:34:07.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:34:07.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:34:07.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:34:07.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:07.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:34:07.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:34:07.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:34:07.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:34:07.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:07.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:34:07.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:34:07.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:07.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:34:07.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:34:07.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:07.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:34:07.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:34:07.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:07.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:34:07.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:34:07.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:34:07.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:34:07.438 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.438 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.438 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.438 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:07.438 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:07.438 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 104.795667482, 2024-04-24T14:34:07.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:34:07.438 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:07.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:34:07.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:07.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:34:07.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:34:07.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:34:07.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:34:07.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:07.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:34:07.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:34:07.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:07.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:34:07.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:34:07.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:07.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:34:07.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:34:07.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:07.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:34:07.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:34:07.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:34:07.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:34:07.444 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.444 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.444 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.444 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:07.444 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:07.444 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 104.79558227, 2024-04-24T14:34:07.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:34:07.444 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.444 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:07.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:07.444 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:34:07.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:07.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:34:07.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:34:07.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:34:07.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:34:07.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:34:07.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:34:07.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:07.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:34:07.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:34:07.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:34:07.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:34:07.449 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.449 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.449 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.450 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:07.450 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:07.450 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 104.795345389, 2024-04-24T14:34:07.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:34:07.450 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.450 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:07.450 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.450 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:07.450 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:07.450 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:34:07.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.451 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:07.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:07.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:34:07.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:07.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:34:07.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:34:07.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:34:07.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:34:07.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:07.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:34:07.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:34:07.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:07.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:34:07.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:34:07.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.455 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 104.795263867, 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:07.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:07.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:34:07.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:07.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:34:07.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:34:07.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:34:07.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:34:07.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:07.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:34:07.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:34:07.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:07.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:34:07.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:34:07.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:07.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:34:07.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:34:07.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:07.461 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:07.462 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 104.36335151999999, 2024-04-24T14:34:07.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:34:07.462 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.462 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:07.462 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.462 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:07.462 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:07.462 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:34:07.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.462 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:07.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:07.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:34:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:34:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:34:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:34:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:34:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:34:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:07.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:34:07.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:34:07.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:07.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:34:07.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:34:07.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:07.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:34:07.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:34:07.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:07.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:34:07.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:34:07.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:34:07.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:34:07.467 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.467 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.467 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 104.36332322, 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:07.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:07.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:34:07.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:34:07.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:34:07.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:34:07.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:34:07.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:34:07.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:07.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:34:07.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:34:07.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:07.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:34:07.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:34:07.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:07.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:34:07.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:34:07.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:07.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:34:07.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:34:07.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:34:07.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:34:07.473 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.474 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.474 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.474 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:07.474 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:07.474 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 104.363235516, 2024-04-24T14:34:07.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:34:07.474 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.474 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:07.474 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.474 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:07.474 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:07.475 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:34:07.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.475 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:07.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:07.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:34:07.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:34:07.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:34:07.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:34:07.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:34:07.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:34:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:34:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:34:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:34:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:34:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:07.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:34:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:34:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:34:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:34:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:34:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:34:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.479 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 104.363112715, 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:07.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:07.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:34:07.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:34:07.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:34:07.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:34:07.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:34:07.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:34:07.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:34:07.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:34:07.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:34:07.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:34:07.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:34:07.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:07.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:34:07.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:34:07.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:07.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:34:07.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:34:07.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:07.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:34:07.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:34:07.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:07.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:34:07.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:34:07.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:34:07.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:34:07.486 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.486 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.486 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.486 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:34:07.486 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:07.486 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 35.056868154999997, 2024-04-24T14:34:07.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:34:07.486 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.486 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:07.486 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.486 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:34:07.486 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:34:07.487 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:34:07.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.487 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:07.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:07.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:34:07.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:07.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:34:07.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:34:07.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:34:07.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:34:07.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:34:07.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:34:07.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:34:07.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:34:07.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:34:07.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:34:07.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:34:07.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:34:07.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:34:07.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:07.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:34:07.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:34:07.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:34:07.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:34:07.491 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.491 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.491 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.491 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.492 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:07.492 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.362744156000002, 2024-04-24T14:34:07.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:34:07.492 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.492 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.492 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.492 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:34:07.492 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:34:07.492 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:34:07.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.493 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:07.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:07.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:34:07.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:07.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:34:07.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:34:07.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:34:07.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:34:07.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:34:07.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:34:07.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:34:07.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:34:07.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:34:07.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:34:07.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:34:07.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:34:07.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:34:07.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:34:07.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:34:07.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:34:07.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:34:07.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:34:07.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:34:07.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:34:07.497 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.497 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.498 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.498 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:07.498 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:07.498 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.4033841900000006, 2024-04-24T14:34:07.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:34:07.498 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.498 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.498 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.498 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:07.498 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:07.498 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:34:07.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:34:07.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:34:07.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:34:07.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:34:07.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:34:07.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:34:07.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:34:07.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:34:07.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:34:07.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:34:07.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:07.503 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:07.504 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.4033036350000003, 2024-04-24T14:34:07.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:34:07.504 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.504 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.504 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.504 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:07.504 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:07.504 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:34:07.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.504 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:07.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:34:07.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:34:07.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:34:07.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:34:07.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:34:07.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:34:07.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:34:07.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:34:07.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:34:07.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:34:07.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:34:07.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:34:07.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:34:07.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:34:07.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:34:07.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:34:07.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:34:07.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:34:07.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:34:07.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:34:07.509 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.509 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.509 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.509 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:07.509 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:07.509 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.4032559060000001, 2024-04-24T14:34:07.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:34:07.509 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.509 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:07.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:07.510 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:34:07.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.510 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:07.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:34:07.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:34:07.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:34:07.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:34:07.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:34:07.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:34:07.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:34:07.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:34:07.514 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.514 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.514 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.4032281429999998, 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:34:07.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:34:07.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:34:07.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:34:07.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:34:07.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:34:07.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:34:07.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:34:07.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:34:07.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:34:07.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:34:07.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:34:07.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:34:07.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:34:07.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:34:07.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:34:07.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:34:07.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:34:07.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:34:07.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:34:07.521 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.521 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.521 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.521 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:07.522 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:07.522 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.4031887080000001, 2024-04-24T14:34:07.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:34:07.522 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.522 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.523 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.523 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:07.523 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:07.523 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:34:07.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.523 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:07.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:07.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:34:07.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:34:07.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:34:07.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:34:07.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:34:07.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:34:07.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:34:07.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:34:07.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:34:07.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:34:07.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:34:07.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:34:07.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:34:07.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:34:07.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:34:07.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:34:07.530 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.531 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.531 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.531 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:07.531 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:07.531 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.4031637240000006, 2024-04-24T14:34:07.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:34:07.531 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.531 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.531 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.531 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:07.531 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:07.531 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:34:07.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.532 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:07.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:07.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:34:07.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:34:07.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:34:07.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:34:07.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:34:07.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:34:07.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:34:07.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:34:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:34:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:34:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:34:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:34:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:34:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:34:07.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:34:07.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:34:07.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:34:07.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:34:07.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:34:07.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:34:07.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:34:07.537 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.537 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.537 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.537 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:07.537 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:07.538 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 8.4031380650000003, 2024-04-24T14:34:07.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:34:07.538 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.538 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.538 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.538 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:07.538 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:07.538 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:34:07.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.538 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:07.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:07.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:34:07.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:34:07.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:34:07.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:07.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:34:07.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:34:07.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:34:07.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:34:07.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:07.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:34:07.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:34:07.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:34:07.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:34:07.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:34:07.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:34:07.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:34:07.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:34:07.545 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.545 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.545 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.545 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.545 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.545 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:07.547 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:07.548 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:07.551+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:07.647 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:07.653+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:07.653 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:07.653 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:07.653 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:07.653 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:07.662 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:07.748 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:07.751+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:07.761 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:07.761 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:07.761 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:07.761 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:07.761 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:07.770 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:07.851 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:07.855+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:07.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T14:34:07.875 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:07.875 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:07.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:07.875 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:07.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.875 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:379 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.875 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:07.875 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.214853895000001, 2024-04-24T14:34:07.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068078410000000002, 2024-04-24T14:34:07.875 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.875 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.875 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.875 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.876 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.876 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 379 2024-04-24T14:34:07.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.876 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:07.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:07.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:07.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664240+0000", 2024-04-24T14:34:07.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6539999999999999e-06 2024-04-24T14:34:07.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664245+0000", 2024-04-24T14:34:07.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2179999999999998e-06 2024-04-24T14:34:07.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664249+0000", 2024-04-24T14:34:07.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2279999999999998e-06 2024-04-24T14:34:07.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670245+0000", 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059962430000000001 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670400+0000", 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015535399999999999 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671046+0000", 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064591599999999996 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.880 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:380 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.880 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:34:07.880 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.214807268000001, 2024-04-24T14:34:07.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076160830000000001, 2024-04-24T14:34:07.880 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.880 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.880 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.880 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.880 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.880 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 380 2024-04-24T14:34:07.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.880 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:34:07.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:34:07.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664284+0000", 2024-04-24T14:34:07.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664287+0000", 2024-04-24T14:34:07.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9739999999999998e-06 2024-04-24T14:34:07.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664302+0000", 2024-04-24T14:34:07.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5214999999999999e-05 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664307+0000", 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6859999999999997e-06 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671098+0000", 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067912479999999997 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671253+0000", 2024-04-24T14:34:07.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015427400000000001 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671901+0000", 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064845600000000005 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.214777723000001, 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:34:07.884 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.885 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.885 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.885 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.885 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.885 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:34:07.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.885 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:07.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:07.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:07.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:34:07.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:34:07.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:34:07.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:34:07.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:34:07.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:34:07.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:34:07.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:34:07.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:34:07.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:34:07.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:34:07.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:34:07.889 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.889 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.889 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.889 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:382 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.889 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:34:07.889 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.211776239999999, 2024-04-24T14:34:07.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067029910000000002, 2024-04-24T14:34:07.889 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.889 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.889 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.889 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.890 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.890 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 382 2024-04-24T14:34:07.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.890 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:34:07.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667315+0000", 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667321+0000", 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2099999999999998e-06 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667336+0000", 2024-04-24T14:34:07.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4612e-05 2024-04-24T14:34:07.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667343+0000", 2024-04-24T14:34:07.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0119999999999999e-06 2024-04-24T14:34:07.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672957+0000", 2024-04-24T14:34:07.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056139700000000002 2024-04-24T14:34:07.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.673127+0000", 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017044599999999999 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674019+0000", 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00089165499999999999 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:07.893 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.211743197000001, 2024-04-24T14:34:07.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:34:07.894 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.894 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.894 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.894 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.894 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.894 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:34:07.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.894 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:07.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:07.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:34:07.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:34:07.898 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.898 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.898 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.898 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.898 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:07.898 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.211719011, 2024-04-24T14:34:07.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:34:07.898 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.899 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.899 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.899 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.899 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.899 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:34:07.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.899 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:07.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:07.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:07.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:34:07.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:34:07.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:34:07.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:34:07.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:34:07.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:34:07.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:34:07.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:34:07.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:34:07.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:34:07.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:34:07.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:34:07.902 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.902 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.902 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.211701627, 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:34:07.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:34:07.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:34:07.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:34:07.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:34:07.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:34:07.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:34:07.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:34:07.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:34:07.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:34:07.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:399 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.197291492000002, 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071317810000000002, 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.907 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 399 2024-04-24T14:34:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:34:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:34:07.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681800+0000", 2024-04-24T14:34:07.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681804+0000", 2024-04-24T14:34:07.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.715e-06 2024-04-24T14:34:07.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681816+0000", 2024-04-24T14:34:07.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1851000000000001e-05 2024-04-24T14:34:07.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681820+0000", 2024-04-24T14:34:07.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6450000000000004e-06 2024-04-24T14:34:07.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688059+0000", 2024-04-24T14:34:07.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062390090000000002 2024-04-24T14:34:07.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688196+0000", 2024-04-24T14:34:07.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013721999999999999 2024-04-24T14:34:07.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688933+0000", 2024-04-24T14:34:07.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073605000000000003 2024-04-24T14:34:07.911 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.911 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.911 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.197265535, 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:34:07.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:34:07.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:34:07.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:34:07.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:34:07.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:34:07.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:34:07.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:34:07.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:34:07.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:34:07.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:34:07.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:34:07.915 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.197233235999999, 2024-04-24T14:34:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:34:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.916 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:34:07.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.917 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:07.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:07.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:34:07.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:34:07.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:34:07.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:34:07.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:34:07.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:34:07.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:07.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:34:07.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:34:07.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:34:07.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:34:07.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:34:07.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:34:07.920 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.920 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.920 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.920 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:414 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.920 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:34:07.921 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.183037826, 2024-04-24T14:34:07.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069877910000000001, 2024-04-24T14:34:07.921 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.921 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.921 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.921 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.921 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.921 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 414 2024-04-24T14:34:07.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.921 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696053+0000", 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696058+0000", 2024-04-24T14:34:07.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7509999999999999e-06 2024-04-24T14:34:07.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696074+0000", 2024-04-24T14:34:07.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5743000000000002e-05 2024-04-24T14:34:07.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696093+0000", 2024-04-24T14:34:07.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9508e-05 2024-04-24T14:34:07.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702314+0000", 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062207010000000004 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702431+0000", 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011723299999999999 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703042+0000", 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000610752 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.924 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.182980958999998, 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:07.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:07.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:34:07.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:34:07.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:34:07.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:34:07.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:34:07.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:34:07.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:34:07.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.182951858999999, 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.930 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.930 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:34:07.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.930 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:07.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:07.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:07.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:34:07.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:34:07.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:34:07.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:34:07.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:34:07.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:34:07.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:34:07.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:34:07.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:34:07.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:34:07.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:34:07.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:34:07.933 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.933 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.933 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.933 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.933 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:07.934 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.182925929, 2024-04-24T14:34:07.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:34:07.934 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.934 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.934 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.934 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.934 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.934 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:34:07.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.934 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:07.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:07.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:07.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:34:07.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:34:07.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:34:07.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:34:07.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:34:07.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:34:07.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:34:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:34:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:34:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:34:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:34:07.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:34:07.938 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.938 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.938 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.938 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:429 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3538944~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.938 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:34:07.938 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.168644488999998, 2024-04-24T14:34:07.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071866789999999996, 2024-04-24T14:34:07.938 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.938 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.938 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.938 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.939 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.939 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 429 2024-04-24T14:34:07.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.939 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:34:07.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:34:07.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710447+0000", 2024-04-24T14:34:07.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710451+0000", 2024-04-24T14:34:07.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5930000000000002e-06 2024-04-24T14:34:07.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710465+0000", 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.411e-05 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710470+0000", 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3129999999999998e-06 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716696+0000", 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006225983 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716901+0000", 2024-04-24T14:34:07.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020550200000000001 2024-04-24T14:34:07.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717634+0000", 2024-04-24T14:34:07.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073292099999999998 2024-04-24T14:34:07.942 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.942 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.942 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.942 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:430 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.942 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:34:07.943 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.167280180999999, 2024-04-24T14:34:07.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068326970000000004, 2024-04-24T14:34:07.943 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.943 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.943 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.943 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.943 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.943 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 430 2024-04-24T14:34:07.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.943 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:34:07.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:34:07.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711810+0000", 2024-04-24T14:34:07.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:07.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711822+0000", 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2166e-05 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711856+0000", 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3816000000000003e-05 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711873+0000", 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6742999999999999e-05 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717693+0000", 2024-04-24T14:34:07.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058204579999999997 2024-04-24T14:34:07.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717864+0000", 2024-04-24T14:34:07.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017128299999999999 2024-04-24T14:34:07.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718645+0000", 2024-04-24T14:34:07.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078033699999999998 2024-04-24T14:34:07.946 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.946 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.947 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.947 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.947 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:07.947 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.167205464999999, 2024-04-24T14:34:07.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:34:07.947 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.947 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:07.947 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.947 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.947 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.947 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:34:07.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.948 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:07.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:07.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:34:07.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:34:07.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:34:07.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:34:07.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:34:07.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:34:07.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:34:07.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:34:07.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:34:07.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:34:07.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.167173533, 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:34:07.951 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.952 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:07.952 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.952 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.952 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.952 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:34:07.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.952 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:07.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:07.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:34:07.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:07.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:34:07.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:34:07.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:34:07.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:34:07.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:34:07.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:34:07.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:34:07.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:34:07.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:34:07.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:34:07.955 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.955 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.955 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.956 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:07.956 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:07.956 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.167106016999998, 2024-04-24T14:34:07.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:34:07.956 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.956 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:07.956 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:07.956 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:07.956 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:07.956 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:34:07.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.956 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:07.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:07.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:34:07.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:07.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:34:07.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:34:07.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:34:07.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:34:07.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:34:07.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:34:07.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:34:07.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:34:07.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:34:07.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:34:07.959 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.960 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.960 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.960 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:34:07.960 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:07.960 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.109693134, 2024-04-24T14:34:07.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:34:07.960 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:07.960 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:07.960 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:07.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:07.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:07.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:07.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:07.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:07.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:07.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:07.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:07.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:07.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:34:07.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:34:07.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:07.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:34:07.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:34:07.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:07.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:34:07.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:34:07.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:07.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:34:07.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:34:07.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:07.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:34:07.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:34:07.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:07.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:34:07.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:34:07.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:07.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:07.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:07.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:34:07.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:34:07.964 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.964 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.964 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.964 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:07.965 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:07.965 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:07.966 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:07.967 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:07.971+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:08.066 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:08.071+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:08.079 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:08.080 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:08.080 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:08.080 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:08.090 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:08.091 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:34:08.091 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:34:08.091 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:08.091 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:08.092 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:08.092 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:08.161 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=18667 2024-04-24T14:34:08.161 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 18667 2024-04-24T14:34:08.161 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 18667 -C rbd-nbd 2024-04-24T14:34:08.167 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:08.167 INFO:tasks.workunit.client.0.smithi173.stdout: 18667 ? 00:00:00 rbd-nbd 2024-04-24T14:34:08.168 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device testrbdnbd16809 18667 2024-04-24T14:34:08.168 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=testrbdnbd16809 2024-04-24T14:34:08.168 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=18667 2024-04-24T14:34:08.168 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd16809 2024-04-24T14:34:08.168 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:08.168 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:08.169 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:08.171+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:08.170 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:08.170 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:08.173 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:08.173 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:08.173 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd16809 2024-04-24T14:34:08.188 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:08.188 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:08.188 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:08.188 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:08.189 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:08.198 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:08.264 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:34:08.264 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^18667\b' 2024-04-24T14:34:08.264 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^18667\b' 2024-04-24T14:34:08.269 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:08.271+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:08.296 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:08.296 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:08.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:08.296 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:08.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.296 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:34:08.297 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:08.297 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.228180510000001, 2024-04-24T14:34:08.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:34:08.297 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.297 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.297 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:08.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:08.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:34:08.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:08.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:34:08.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:34:08.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:34:08.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:34:08.301 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.301 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.301 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.301 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:34:08.301 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:08.301 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 14.233365931, 2024-04-24T14:34:08.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:34:08.301 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.301 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.301 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:08.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:08.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:34:08.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:08.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:34:08.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:34:08.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:34:08.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:34:08.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:34:08.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:34:08.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:34:08.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:34:08.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:34:08.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.306 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:34:08.306 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:08.306 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 14.222436673000001, 2024-04-24T14:34:08.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:34:08.306 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.306 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.306 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:08.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:08.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:34:08.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:08.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:34:08.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:34:08.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:34:08.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:34:08.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:34:08.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:34:08.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:34:08.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:34:08.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:34:08.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:34:08.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:34:08.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:34:08.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:34:08.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:34:08.310 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.310 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.310 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.310 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:34:08.310 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:08.314 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 14.211480591999999, 2024-04-24T14:34:08.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:34:08.314 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.314 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.314 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:08.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:08.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:34:08.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:08.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:34:08.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:34:08.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:34:08.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:34:08.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:34:08.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:34:08.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:34:08.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:34:08.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:34:08.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:34:08.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:34:08.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:34:08.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:34:08.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:34:08.320 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.320 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.320 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.320 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:34:08.320 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:08.320 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 14.199911226999999, 2024-04-24T14:34:08.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:34:08.321 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.321 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.321 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:08.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:08.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:34:08.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:08.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:34:08.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:34:08.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:34:08.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:34:08.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:34:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:34:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:34:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:34:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:34:08.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:34:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:34:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:34:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:34:08.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:34:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:109 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'60, mlcod=62'58)", 2024-04-24T14:34:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:34:08.326 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.956799062, 2024-04-24T14:34:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036290020000000001, 2024-04-24T14:34:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:34:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:34:08.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343311+0000", 2024-04-24T14:34:08.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:08.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343841+0000", 2024-04-24T14:34:08.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053003399999999997 2024-04-24T14:34:08.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343876+0000", 2024-04-24T14:34:08.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4479e-05 2024-04-24T14:34:08.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343888+0000", 2024-04-24T14:34:08.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1623e-05 2024-04-24T14:34:08.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343996+0000", 2024-04-24T14:34:08.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108028 2024-04-24T14:34:08.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.344044+0000", 2024-04-24T14:34:08.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8081999999999998e-05 2024-04-24T14:34:08.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.346872+0000", 2024-04-24T14:34:08.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028280789999999998 2024-04-24T14:34:08.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.346942+0000", 2024-04-24T14:34:08.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0405000000000001e-05 2024-04-24T14:34:08.331 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.332 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.332 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.332 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:34:08.332 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:08.332 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.946791568, 2024-04-24T14:34:08.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:34:08.332 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.333 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.333 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:08.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:08.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:34:08.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:08.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:34:08.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:34:08.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:34:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:34:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:34:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:34:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:34:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:34:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:34:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:34:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:34:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:34:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:34:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:34:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:34:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.9360464, 2024-04-24T14:34:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:34:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:34:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:34:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:34:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:34:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:34:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:34:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:34:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:34:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:34:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:34:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:34:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:34:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:34:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:34:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:34:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:34:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.923680124000001, 2024-04-24T14:34:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:34:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:34:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:34:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:34:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:34:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:34:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:34:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:34:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:34:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:34:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:34:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:34:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:34:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:34:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:34:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:34:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:34:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.798744002999999, 2024-04-24T14:34:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:34:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:34:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:34:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:34:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:34:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:34:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:34:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:34:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:34:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:34:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:34:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:34:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:34:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:34:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:34:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:34:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:138 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'70, mlcod=62'68)", 2024-04-24T14:34:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:34:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.788357849, 2024-04-24T14:34:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036613330000000001, 2024-04-24T14:34:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:34:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:34:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511753+0000", 2024-04-24T14:34:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512370+0000", 2024-04-24T14:34:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061776500000000005 2024-04-24T14:34:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512395+0000", 2024-04-24T14:34:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4549000000000002e-05 2024-04-24T14:34:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512404+0000", 2024-04-24T14:34:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1130000000000007e-06 2024-04-24T14:34:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512517+0000", 2024-04-24T14:34:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112679 2024-04-24T14:34:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512581+0000", 2024-04-24T14:34:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4274000000000002e-05 2024-04-24T14:34:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.515303+0000", 2024-04-24T14:34:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027220640000000002 2024-04-24T14:34:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.515416+0000", 2024-04-24T14:34:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112644 2024-04-24T14:34:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:34:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.779356160000001, 2024-04-24T14:34:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:34:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:34:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:34:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:34:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:34:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:34:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:34:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:34:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:34:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:34:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:34:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:34:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:34:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:34:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:34:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:34:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:34:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.769628317, 2024-04-24T14:34:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:34:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:34:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:34:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:34:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:34:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:34:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:34:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:34:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:34:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:34:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:34:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:34:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:34:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:34:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:34:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:34:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:172 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'76, mlcod=62'74)", 2024-04-24T14:34:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:34:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.619360963, 2024-04-24T14:34:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032080400000000001, 2024-04-24T14:34:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:34:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:34:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680750+0000", 2024-04-24T14:34:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681277+0000", 2024-04-24T14:34:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052776400000000001 2024-04-24T14:34:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681304+0000", 2024-04-24T14:34:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6948e-05 2024-04-24T14:34:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681315+0000", 2024-04-24T14:34:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0648e-05 2024-04-24T14:34:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681413+0000", 2024-04-24T14:34:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7926999999999999e-05 2024-04-24T14:34:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681470+0000", 2024-04-24T14:34:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7401999999999997e-05 2024-04-24T14:34:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.683923+0000", 2024-04-24T14:34:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024526790000000001 2024-04-24T14:34:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.683959+0000", 2024-04-24T14:34:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6327e-05 2024-04-24T14:34:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:34:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.609518637000001, 2024-04-24T14:34:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:34:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:34:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:34:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:34:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:34:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:34:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:34:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:34:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:34:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:34:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:34:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:34:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:34:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:34:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:34:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:34:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:34:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.599789640999999, 2024-04-24T14:34:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:34:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:34:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:34:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:34:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:34:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:34:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:34:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:34:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:34:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:34:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:34:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:34:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:34:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:34:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:34:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:34:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:178 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'82, mlcod=62'80)", 2024-04-24T14:34:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:34:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.589322265, 2024-04-24T14:34:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029527640000000001, 2024-04-24T14:34:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:34:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:34:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710789+0000", 2024-04-24T14:34:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711538+0000", 2024-04-24T14:34:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00074893099999999999 2024-04-24T14:34:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711565+0000", 2024-04-24T14:34:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7526999999999999e-05 2024-04-24T14:34:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711576+0000", 2024-04-24T14:34:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0553999999999999e-05 2024-04-24T14:34:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711633+0000", 2024-04-24T14:34:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7258e-05 2024-04-24T14:34:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711653+0000", 2024-04-24T14:34:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0460999999999999e-05 2024-04-24T14:34:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.713712+0000", 2024-04-24T14:34:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020587790000000002 2024-04-24T14:34:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.713743+0000", 2024-04-24T14:34:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0519999999999999e-05 2024-04-24T14:34:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:34:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 9.2648188059999992, 2024-04-24T14:34:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:34:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:34:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:34:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:34:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:34:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:34:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:34:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:34:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:34:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:34:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:34:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:34:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:34:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:34:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:34:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.4206680580000004, 2024-04-24T14:34:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:34:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:34:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:34:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:34:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:34:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:34:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:34:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:34:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:34:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:34:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:34:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.957086+0000", 2024-04-24T14:34:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 5.3430267130000004, 2024-04-24T14:34:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029645230000000002, 2024-04-24T14:34:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957086+0000", 2024-04-24T14:34:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957086+0000", 2024-04-24T14:34:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957084+0000", 2024-04-24T14:34:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957106+0000", 2024-04-24T14:34:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1866e-05 2024-04-24T14:34:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957130+0000", 2024-04-24T14:34:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4266000000000001e-05 2024-04-24T14:34:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957140+0000", 2024-04-24T14:34:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.007e-05 2024-04-24T14:34:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957264+0000", 2024-04-24T14:34:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012356699999999999 2024-04-24T14:34:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.960050+0000", 2024-04-24T14:34:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027863570000000002 2024-04-24T14:34:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:08.410 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:08.413 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:08.413 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *18667\b' 2024-04-24T14:34:08.413 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *18667\b' 2024-04-24T14:34:08.413 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:34:08.413 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:08.413 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:34:08.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:08.414 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:08.417+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:08.513 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:08.515+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:08.524 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:08.524 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:08.524 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:08.524 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:08.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:08.614 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:08.619+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:08.637 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:08.637 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:08.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:08.715 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:08.721+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:08.797 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.9215320870000001, 2024-04-24T14:34:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:34:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:08.798 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:34:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:34:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:34:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:34:08.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:34:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:34:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:34:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:34:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:34:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:08.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:34:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:34:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:34:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:34:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:34:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:34:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:34:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:34:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:34:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:34:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:34:08.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:34:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.921503102, 2024-04-24T14:34:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:34:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:08.804 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:34:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:34:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:34:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:34:08.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:34:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:34:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:34:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:34:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:34:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:08.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:34:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:34:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:34:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:34:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:34:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:34:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:34:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:34:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:34:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:34:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:34:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:34:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.921484617, 2024-04-24T14:34:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:34:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.810 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:34:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:08.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:34:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:34:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:34:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:34:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:34:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:34:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:34:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:34:08.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:34:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:34:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:34:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:34:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:34:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:08.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:34:08.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:34:08.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:34:08.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:34:08.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:34:08.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:34:08.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:34:08.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:34:08.816 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.816 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.816 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.816 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:08.816 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:08.816 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.9214693360000004, 2024-04-24T14:34:08.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:34:08.816 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.816 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.817 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:08.817 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:08.817 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:08.817 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:34:08.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.817 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:08.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:08.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:08.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:08.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:34:08.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:34:08.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:34:08.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:08.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:34:08.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:34:08.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:34:08.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:34:08.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:08.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:34:08.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:34:08.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:34:08.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:34:08.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:34:08.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:34:08.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:34:08.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:34:08.822 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.822 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.822 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.822 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:08.822 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:08.822 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.9214535550000003, 2024-04-24T14:34:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:34:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:34:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:08.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:34:08.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:34:08.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:34:08.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:34:08.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:34:08.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:34:08.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:34:08.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:34:08.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:08.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:34:08.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:34:08.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:34:08.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:34:08.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:08.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:34:08.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:34:08.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:34:08.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:34:08.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:34:08.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:34:08.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:34:08.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:34:08.828 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.828 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.828 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.828 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:08.828 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:08.828 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.9214392670000002, 2024-04-24T14:34:08.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:08.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:08.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:08.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:34:08.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:34:08.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:34:08.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:34:08.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:34:08.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:34:08.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:34:08.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:34:08.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:08.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:34:08.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:34:08.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:34:08.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:34:08.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:08.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:34:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:34:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:34:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:34:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:34:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:34:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:34:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:34:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.9214195150000002, 2024-04-24T14:34:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:34:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:34:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:34:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:34:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:34:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:34:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:34:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:34:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:34:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:34:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:34:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:34:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:34:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:34:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.9214020679999999, 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:08.840 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:34:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:34:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:34:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:08.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:34:08.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:34:08.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:34:08.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:34:08.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:08.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:34:08.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:34:08.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:34:08.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:34:08.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:34:08.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:34:08.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:34:08.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:34:08.846 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.846 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.846 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.846 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4d9853ae:::rbd_id.INVALIDIMAGE:head, version: 53'12, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4d9853ae:::rbd_id.INVALIDIMAGE:head@53'12, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:08.846 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:08.846 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.8961235370000002, 2024-04-24T14:34:08.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075925020000000001, 2024-04-24T14:34:08.846 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.847 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.847 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:08.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:08.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905243+0000", 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905253+0000", 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1619999999999997e-06 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905262+0000", 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2399999999999996e-06 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905268+0000", 2024-04-24T14:34:08.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2530000000000001e-06 2024-04-24T14:34:08.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912457+0000", 2024-04-24T14:34:08.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071890779999999998 2024-04-24T14:34:08.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912462+0000", 2024-04-24T14:34:08.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5159999999999996e-06 2024-04-24T14:34:08.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912836+0000", 2024-04-24T14:34:08.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037484699999999998 2024-04-24T14:34:08.850 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.850 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.850 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.850 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 62'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@62'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.8725046079999998, 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036829689999999999, 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:08.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928862+0000", 2024-04-24T14:34:08.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:08.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930264+0000", 2024-04-24T14:34:08.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014016059999999999 2024-04-24T14:34:08.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930281+0000", 2024-04-24T14:34:08.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7859999999999998e-05 2024-04-24T14:34:08.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930291+0000", 2024-04-24T14:34:08.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9299999999999998e-06 2024-04-24T14:34:08.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930894+0000", 2024-04-24T14:34:08.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060250899999999999 2024-04-24T14:34:08.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930903+0000", 2024-04-24T14:34:08.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4599999999999992e-06 2024-04-24T14:34:08.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.932546+0000", 2024-04-24T14:34:08.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016425019999999999 2024-04-24T14:34:08.854 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.854 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.854 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.855 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head, version: 62'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head@62'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:08.855 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:08.855 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.8667896600000002, 2024-04-24T14:34:08.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037783690000000002, 2024-04-24T14:34:08.855 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.855 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.855 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:08.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:08.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934576+0000", 2024-04-24T14:34:08.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:08.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936033+0000", 2024-04-24T14:34:08.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001456658 2024-04-24T14:34:08.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936071+0000", 2024-04-24T14:34:08.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7752e-05 2024-04-24T14:34:08.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936087+0000", 2024-04-24T14:34:08.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5919000000000001e-05 2024-04-24T14:34:08.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936226+0000", 2024-04-24T14:34:08.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013910800000000001 2024-04-24T14:34:08.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936240+0000", 2024-04-24T14:34:08.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.456e-05 2024-04-24T14:34:08.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938356+0000", 2024-04-24T14:34:08.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021157540000000001 2024-04-24T14:34:08.858 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.858 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.858 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.859 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 62'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@62'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:08.859 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:08.859 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.8657431320000004, 2024-04-24T14:34:08.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052678259999999998, 2024-04-24T14:34:08.859 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.859 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.859 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:08.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935623+0000", 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937055+0000", 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014314480000000001 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937085+0000", 2024-04-24T14:34:08.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0204000000000001e-05 2024-04-24T14:34:08.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937097+0000", 2024-04-24T14:34:08.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1708e-05 2024-04-24T14:34:08.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938410+0000", 2024-04-24T14:34:08.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013128910000000001 2024-04-24T14:34:08.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938419+0000", 2024-04-24T14:34:08.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1740000000000006e-06 2024-04-24T14:34:08.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.940892+0000", 2024-04-24T14:34:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024734990000000001 2024-04-24T14:34:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 62'82, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@62'82, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.8562317410000002, 2024-04-24T14:34:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003877447, 2024-04-24T14:34:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.863 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.864 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:08.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:08.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945134+0000", 2024-04-24T14:34:08.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:08.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946323+0000", 2024-04-24T14:34:08.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001188635 2024-04-24T14:34:08.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946352+0000", 2024-04-24T14:34:08.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9238e-05 2024-04-24T14:34:08.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946374+0000", 2024-04-24T14:34:08.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1396999999999999e-05 2024-04-24T14:34:08.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.947098+0000", 2024-04-24T14:34:08.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072458199999999998 2024-04-24T14:34:08.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.947109+0000", 2024-04-24T14:34:08.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0557000000000001e-05 2024-04-24T14:34:08.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.949013+0000", 2024-04-24T14:34:08.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019044929999999999 2024-04-24T14:34:08.867 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.867 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.867 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.867 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:08.867 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:08.867 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.8497747560000004, 2024-04-24T14:34:08.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041579369999999996, 2024-04-24T14:34:08.868 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.868 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.868 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:08.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:08.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951585+0000", 2024-04-24T14:34:08.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999928 2024-04-24T14:34:08.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953339+0000", 2024-04-24T14:34:08.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017540019999999999 2024-04-24T14:34:08.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953363+0000", 2024-04-24T14:34:08.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3178000000000001e-05 2024-04-24T14:34:08.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953374+0000", 2024-04-24T14:34:08.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1015e-05 2024-04-24T14:34:08.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953522+0000", 2024-04-24T14:34:08.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014857800000000001 2024-04-24T14:34:08.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953565+0000", 2024-04-24T14:34:08.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2511e-05 2024-04-24T14:34:08.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955751+0000", 2024-04-24T14:34:08.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002185958 2024-04-24T14:34:08.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.871 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 62'82, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@62'82, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.8493602400000002, 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005349749, 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952006+0000", 2024-04-24T14:34:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953689+0000", 2024-04-24T14:34:08.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00168326 2024-04-24T14:34:08.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953717+0000", 2024-04-24T14:34:08.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7994e-05 2024-04-24T14:34:08.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953746+0000", 2024-04-24T14:34:08.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8059999999999999e-05 2024-04-24T14:34:08.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955798+0000", 2024-04-24T14:34:08.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002052089 2024-04-24T14:34:08.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955812+0000", 2024-04-24T14:34:08.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2024-04-24T14:34:08.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.957357+0000", 2024-04-24T14:34:08.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001545537 2024-04-24T14:34:08.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.876 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.876 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:08.876 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:08.876 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.8405701219999999, 2024-04-24T14:34:08.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051567999999999996, 2024-04-24T14:34:08.877 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.877 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.877 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:08.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:08.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960796+0000", 2024-04-24T14:34:08.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:08.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962459+0000", 2024-04-24T14:34:08.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016622259999999999 2024-04-24T14:34:08.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962481+0000", 2024-04-24T14:34:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2359999999999999e-05 2024-04-24T14:34:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962496+0000", 2024-04-24T14:34:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.523e-05 2024-04-24T14:34:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.964299+0000", 2024-04-24T14:34:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018025129999999999 2024-04-24T14:34:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.964316+0000", 2024-04-24T14:34:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7436999999999999e-05 2024-04-24T14:34:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.965954+0000", 2024-04-24T14:34:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016381099999999999 2024-04-24T14:34:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head, version: 62'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head@62'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.807563933, 2024-04-24T14:34:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003994821, 2024-04-24T14:34:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:08.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993802+0000", 2024-04-24T14:34:08.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:08.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994846+0000", 2024-04-24T14:34:08.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00104368 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994884+0000", 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7796000000000002e-05 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994900+0000", 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5917e-05 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995046+0000", 2024-04-24T14:34:08.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014653499999999999 2024-04-24T14:34:08.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995055+0000", 2024-04-24T14:34:08.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0759999999999992e-06 2024-04-24T14:34:08.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.997798+0000", 2024-04-24T14:34:08.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027432709999999998 2024-04-24T14:34:08.884 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(2.2 68/67 [PushOp(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 49'1, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@49'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.807256475, 2024-04-24T14:34:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038339310000000001, 2024-04-24T14:34:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994110+0000", 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994914+0000", 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080383099999999997 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994943+0000", 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9475000000000002e-05 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994955+0000", 2024-04-24T14:34:08.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1525000000000001e-05 2024-04-24T14:34:08.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995123+0000", 2024-04-24T14:34:08.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016812899999999999 2024-04-24T14:34:08.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995143+0000", 2024-04-24T14:34:08.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9724999999999998e-05 2024-04-24T14:34:08.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.997945+0000", 2024-04-24T14:34:08.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028023890000000002 2024-04-24T14:34:08.889 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.889 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.889 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.889 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head, version: 62'144, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head@62'144, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:08.889 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:08.889 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.7669512799999998, 2024-04-24T14:34:08.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041651270000000002, 2024-04-24T14:34:08.889 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.890 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.890 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:08.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:08.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034415+0000", 2024-04-24T14:34:08.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:08.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035025+0000", 2024-04-24T14:34:08.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060999700000000003 2024-04-24T14:34:08.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035077+0000", 2024-04-24T14:34:08.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2132999999999997e-05 2024-04-24T14:34:08.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035100+0000", 2024-04-24T14:34:08.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2690000000000001e-05 2024-04-24T14:34:08.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035293+0000", 2024-04-24T14:34:08.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019358900000000001 2024-04-24T14:34:08.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035311+0000", 2024-04-24T14:34:08.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7952999999999999e-05 2024-04-24T14:34:08.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.038581+0000", 2024-04-24T14:34:08.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032700670000000002 2024-04-24T14:34:08.893 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.893 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.893 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.893 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head, version: 62'144, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head@62'144, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:08.894 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:08.894 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.752785845, 2024-04-24T14:34:08.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039835970000000002, 2024-04-24T14:34:08.894 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:08.894 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:08.894 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:08.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:08.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:08.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:08.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:08.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:08.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:08.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048581+0000", 2024-04-24T14:34:08.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:08.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:08.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049792+0000", 2024-04-24T14:34:08.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012111579999999999 2024-04-24T14:34:08.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:08.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049823+0000", 2024-04-24T14:34:08.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1137999999999999e-05 2024-04-24T14:34:08.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:08.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049838+0000", 2024-04-24T14:34:08.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5061e-05 2024-04-24T14:34:08.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:08.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.050005+0000", 2024-04-24T14:34:08.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016745400000000001 2024-04-24T14:34:08.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:08.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.050022+0000", 2024-04-24T14:34:08.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7022000000000001e-05 2024-04-24T14:34:08.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:08.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:08.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:08.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.052565+0000", 2024-04-24T14:34:08.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025428669999999999 2024-04-24T14:34:08.897 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.897 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.897 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.897 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:08.897 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:08.897 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:08.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:34:08.899 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:34:08.899 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd config global set global rbd_default_pool new_default_pool16809 2024-04-24T14:34:08.899 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd create --size 10M testrbdnbd16809 2024-04-24T14:34:08.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:08.901 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-24T14:34:08.902 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:08.903+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:08.955 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd16809 2024-04-24T14:34:08.955 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:34:08.955 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:34:08.958 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:34:08.958 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:34:08.960 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:34:08.960 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:34:08.960 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd16809 2024-04-24T14:34:09.000 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:09.005+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:09.007 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:09.007 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:09.007 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:09.007 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:09.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:09.031 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:09.051 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:09.069 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:09.090 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:09.101 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:09.105+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:09.111 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:09.112 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:09.112 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:09.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:09.130 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:34:09.130 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid new_default_pool16809 2024-04-24T14:34:09.130 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=new_default_pool16809 2024-04-24T14:34:09.130 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:09.133 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:09.133 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''new_default_pool16809'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:09.135 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:09.137 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:09.138 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:09.202 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:09.207+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:09.203 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=18795 2024-04-24T14:34:09.203 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 18795 2024-04-24T14:34:09.203 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 18795 -C rbd-nbd 2024-04-24T14:34:09.211 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:09.211 INFO:tasks.workunit.client.0.smithi173.stdout: 18795 ? 00:00:00 rbd-nbd 2024-04-24T14:34:09.212 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device testrbdnbd16809 18795 2024-04-24T14:34:09.212 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=testrbdnbd16809 2024-04-24T14:34:09.213 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=18795 2024-04-24T14:34:09.213 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd16809 2024-04-24T14:34:09.213 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:09.213 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:09.214 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:09.214 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:09.216 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:09.216 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:09.217 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:09.217 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:09.217 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd16809 2024-04-24T14:34:09.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:09.302 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:09.307+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:09.309 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:34:09.309 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^18795\b' 2024-04-24T14:34:09.309 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^18795\b' 2024-04-24T14:34:09.324 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:09.324 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:09.333 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:09.362 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:09.363 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:34:09.363 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *18795\b' 2024-04-24T14:34:09.364 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *18795\b' 2024-04-24T14:34:09.377 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:09.377 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:34:09.403 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:09.407+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:09.432 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:09.432 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:09.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:09.504 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:09.509+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:09.590 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:09.590 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:09.599 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:09.643 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:09.649+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:09.706 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:09.711+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:09.710 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:09.710 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:09.719 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:09.807 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:09.811+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:09.824 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:09.824 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:09.834 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:09.879 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:34:09.879 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd config global rm global rbd_default_pool 2024-04-24T14:34:09.908 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:09.911+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:09.927 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:09.927 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:09.934 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:34:09.935 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:34:09.935 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:34:09.937 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:34:09.938 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:09.939 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:34:09.940 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:34:09.940 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:34:09.940 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:34:10.009 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:10.011+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:10.032 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:10.032 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:10.041 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:10.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:34:10.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:34:10.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:10.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:10.086 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:10.086 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:10.109 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:10.113+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:10.145 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:10.145 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:10.155 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:10.160 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=18891 2024-04-24T14:34:10.161 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 18891 2024-04-24T14:34:10.161 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 18891 -C rbd-nbd 2024-04-24T14:34:10.174 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:10.174 INFO:tasks.workunit.client.0.smithi173.stdout: 18891 ? 00:00:00 rbd-nbd 2024-04-24T14:34:10.174 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo kill 18891 2024-04-24T14:34:10.174 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:10.175 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:10.176 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:10.177 INFO:tasks.workunit.client.0.smithi173.stderr:++ which kill 2024-04-24T14:34:10.179 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/kill 2024-04-24T14:34:10.179 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:10.179 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/kill 18891 2024-04-24T14:34:10.194 INFO:tasks.workunit.client.0.smithi173.stderr:++ seq 10 2024-04-24T14:34:10.202 INFO:tasks.workunit.client.0.smithi173.stderr:+ for i in `seq 10` 2024-04-24T14:34:10.203 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:34:10.203 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^18891 *rbd *testrbdnbd16809' 2024-04-24T14:34:10.203 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^18891 *rbd *testrbdnbd16809' 2024-04-24T14:34:10.211 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:10.217+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:10.264 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:10.264 INFO:tasks.workunit.client.0.smithi173.stderr:+ break 2024-04-24T14:34:10.265 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^18891 *rbd *testrbdnbd16809' 2024-04-24T14:34:10.265 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:34:10.265 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^18891 *rbd *testrbdnbd16809' 2024-04-24T14:34:10.300 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:10.300 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:10.309 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:10.311 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:10.315+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:10.323 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:10.323 INFO:tasks.workunit.client.0.smithi173.stderr:+ QUIESCE_HOOK=/tmp/tmp.waDYLkcsIV/quiesce.sh 2024-04-24T14:34:10.323 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.waDYLkcsIV/quiesce.sh rbd/testrbdnbd16809 2024-04-24T14:34:10.323 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:34:10.324 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:34:10.326 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:34:10.326 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:34:10.327 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:34:10.327 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:34:10.328 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.waDYLkcsIV/quiesce.sh rbd/testrbdnbd16809 2024-04-24T14:34:10.408 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:10.408 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:10.412 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:10.417+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:10.418 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:10.505 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:34:10.505 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:34:10.505 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:10.505 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:10.505 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:10.506 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:10.512 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:10.515+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:10.563 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:10.563 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:10.572 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:10.584 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=18955 2024-04-24T14:34:10.584 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 18955 2024-04-24T14:34:10.584 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 18955 -C rbd-nbd 2024-04-24T14:34:10.595 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:10.595 INFO:tasks.workunit.client.0.smithi173.stdout: 18955 ? 00:00:00 rbd-nbd 2024-04-24T14:34:10.596 INFO:tasks.workunit.client.0.smithi173.stderr:+ test '!' -e /tmp/tmp.waDYLkcsIV/quiesce.sh 2024-04-24T14:34:10.596 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false rbd snap create rbd/testrbdnbd16809@quiesce1 2024-04-24T14:34:10.597 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd snap create rbd/testrbdnbd16809@quiesce1 2024-04-24T14:34:10.613 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:10.617+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:10.672 INFO:tasks.workunit.client.0.smithi173.stderr: Creating snap: 10% complete...2024-04-24T14:34:10.673+0000 7f8ba3fff700 -1 librbd::ImageWatcher: 0x7f8b98007b10 failed to notify quiesce: (1) Operation not permitted 2024-04-24T14:34:10.672 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:34:10.673+0000 7f8ba3fff700 -1 librbd::SnapshotCreateRequest: failed to notify quiesce: (1) Operation not permitted 2024-04-24T14:34:10.674 INFO:tasks.workunit.client.0.smithi173.stderr: Creating snap: 10% complete...failed. 2024-04-24T14:34:10.674 INFO:tasks.workunit.client.0.smithi173.stderr:rbd: failed to create snapshot: (1) Operation not permitted 2024-04-24T14:34:10.674 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:10.675 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:10.684 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:10.691 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:10.691 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T14:34:10.691 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:10.691 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:10.693 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:10.694 INFO:tasks.workunit.client.0.smithi173.stderr:++ which dd 2024-04-24T14:34:10.696 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/dd 2024-04-24T14:34:10.696 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:10.696 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T14:34:10.730 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:10.735+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:10.739 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records in 2024-04-24T14:34:10.739 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records out 2024-04-24T14:34:10.740 INFO:tasks.workunit.client.0.smithi173.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359736 s, 29.1 MB/s 2024-04-24T14:34:10.741 INFO:tasks.workunit.client.0.smithi173.stderr:+ touch /tmp/tmp.waDYLkcsIV/quiesce.sh 2024-04-24T14:34:10.742 INFO:tasks.workunit.client.0.smithi173.stderr:+ chmod +x /tmp/tmp.waDYLkcsIV/quiesce.sh 2024-04-24T14:34:10.744 INFO:tasks.workunit.client.0.smithi173.stderr:+ cat 2024-04-24T14:34:10.746 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd snap create rbd/testrbdnbd16809@quiesce1 2024-04-24T14:34:10.811 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:10.811 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:10.815 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:10.819+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:10.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:10.860 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 1] 2024-04-24T14:34:10.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T14:34:10.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T14:34:10.915 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:10.919+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:10.919 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:10.919 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:10.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:10.947 INFO:tasks.workunit.client.0.smithi173.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-04-24T14:34:10.958 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T14:34:10.958 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:10.958 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:10.960 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:10.961 INFO:tasks.workunit.client.0.smithi173.stderr:++ which dd 2024-04-24T14:34:10.962 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/dd 2024-04-24T14:34:10.963 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:10.963 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T14:34:10.993 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records in 2024-04-24T14:34:10.993 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records out 2024-04-24T14:34:10.993 INFO:tasks.workunit.client.0.smithi173.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021358 s, 49.1 MB/s 2024-04-24T14:34:10.995 INFO:tasks.workunit.client.0.smithi173.stderr:++ cat /tmp/tmp.waDYLkcsIV/quiesce 2024-04-24T14:34:10.997 INFO:tasks.workunit.client.0.smithi173.stderr:+ test /dev/nbd0 = /dev/nbd0 2024-04-24T14:34:10.998 INFO:tasks.workunit.client.0.smithi173.stderr:++ cat /tmp/tmp.waDYLkcsIV/unquiesce 2024-04-24T14:34:10.999 INFO:tasks.workunit.client.0.smithi173.stderr:+ test /dev/nbd0 = /dev/nbd0 2024-04-24T14:34:11.000 INFO:tasks.workunit.client.0.smithi173.stderr:+ touch /tmp/tmp.waDYLkcsIV/quiesce.sh 2024-04-24T14:34:11.001 INFO:tasks.workunit.client.0.smithi173.stderr:+ chmod +x /tmp/tmp.waDYLkcsIV/quiesce.sh 2024-04-24T14:34:11.002 INFO:tasks.workunit.client.0.smithi173.stderr:+ cat 2024-04-24T14:34:11.005 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false rbd snap create rbd/testrbdnbd16809@quiesce2 2024-04-24T14:34:11.005 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd snap create rbd/testrbdnbd16809@quiesce2 2024-04-24T14:34:11.016 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:11.019+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:11.076 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:11.078 INFO:tasks.workunit.client.0.smithi173.stderr: Creating snap: 10% complete... Creating snap: 10% complete...failed. 2024-04-24T14:34:11.078 INFO:tasks.workunit.client.0.smithi173.stderr:rbd: failed to create snapshot: (22) Invalid argument 2024-04-24T14:34:11.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:11.088 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:11.088 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T14:34:11.089 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:11.089 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:11.090 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:11.090 INFO:tasks.workunit.client.0.smithi173.stderr:++ which dd 2024-04-24T14:34:11.093 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/dd 2024-04-24T14:34:11.093 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:11.093 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T14:34:11.116 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records in 2024-04-24T14:34:11.117 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records out 2024-04-24T14:34:11.117 INFO:tasks.workunit.client.0.smithi173.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132565 s, 79.1 MB/s 2024-04-24T14:34:11.117 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:11.121+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:11.118 INFO:tasks.workunit.client.0.smithi173.stderr:+ cat 2024-04-24T14:34:11.120 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd snap create rbd/testrbdnbd16809@quiesce2 2024-04-24T14:34:11.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:34:11.215 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:11.215 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:11.217 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:11.219+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:11.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:11.295 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:11.316 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:11.317 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:11.323+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:11.326 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:11.326 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:11.326 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:11.326 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:11.326 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:11.335 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:11.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:11.357 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:11.393 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:11.418 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:11.419 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:11.423+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:11.421 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:11.441 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:11.441 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:11.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:11.441 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:11.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.442 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:97 3.1 3:990e5750:::rbd_header.12ff64619e2d:head [watch reconnect cookie 140276853203408 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:11.442 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:11.442 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.093256886999999, 2024-04-24T14:34:11.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012161632, 2024-04-24T14:34:11.442 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.442 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.442 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.442 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.442 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.443 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 97 2024-04-24T14:34:11.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.443 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:11.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:11.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352830+0000", 2024-04-24T14:34:11.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352834+0000", 2024-04-24T14:34:11.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2699999999999998e-06 2024-04-24T14:34:11.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352844+0000", 2024-04-24T14:34:11.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0406e-05 2024-04-24T14:34:11.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352850+0000", 2024-04-24T14:34:11.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6439999999999997e-06 2024-04-24T14:34:11.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362391+0000", 2024-04-24T14:34:11.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095408969999999996 2024-04-24T14:34:11.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362683+0000", 2024-04-24T14:34:11.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029238000000000002 2024-04-24T14:34:11.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.364992+0000", 2024-04-24T14:34:11.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023092020000000002 2024-04-24T14:34:11.447 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.447 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.447 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.447 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:225 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:11.447 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:11.447 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.940814775, 2024-04-24T14:34:11.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010959015000000001, 2024-04-24T14:34:11.447 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.448 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.448 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.448 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.448 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.448 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 225 2024-04-24T14:34:11.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.448 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:11.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:11.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:11.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:11.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505275+0000", 2024-04-24T14:34:11.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0269999999999998e-06 2024-04-24T14:34:11.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505279+0000", 2024-04-24T14:34:11.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4279999999999998e-06 2024-04-24T14:34:11.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505282+0000", 2024-04-24T14:34:11.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.982e-06 2024-04-24T14:34:11.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515209+0000", 2024-04-24T14:34:11.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099266419999999994 2024-04-24T14:34:11.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515466+0000", 2024-04-24T14:34:11.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025719700000000002 2024-04-24T14:34:11.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.516232+0000", 2024-04-24T14:34:11.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076608899999999996 2024-04-24T14:34:11.452 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.453 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.453 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.453 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:11.453 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.689783033000001, 2024-04-24T14:34:11.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:34:11.453 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.453 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.453 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.453 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.454 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:34:11.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.454 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:11.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:11.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:34:11.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:11.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:34:11.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:34:11.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:34:11.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:34:11.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:34:11.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:34:11.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:34:11.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:34:11.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:11.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:34:11.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:34:11.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:34:11.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:34:11.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:11.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:34:11.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:34:11.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:34:11.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:34:11.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:34:11.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:34:11.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:34:11.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:34:11.460 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.460 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.460 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.461 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.461 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:11.461 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.689732199000002, 2024-04-24T14:34:11.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:34:11.461 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.461 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.461 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.461 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.461 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.461 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:34:11.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.462 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:11.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:11.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:11.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:11.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:34:11.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:34:11.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:34:11.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:34:11.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:34:11.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:34:11.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:34:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:34:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:34:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:34:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:34:11.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:34:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:34:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:34:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:34:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:34:11.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:34:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:34:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:34:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:34:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.467 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.689695892, 2024-04-24T14:34:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:34:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.468 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:34:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:34:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:34:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:34:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:34:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:34:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:34:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:34:11.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:34:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:34:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:34:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:34:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:34:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:34:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:34:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:34:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:34:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:34:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:11.475 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.686284642, 2024-04-24T14:34:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:34:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:34:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:11.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:11.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:34:11.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:11.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:34:11.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:34:11.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:34:11.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:34:11.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:34:11.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:34:11.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:34:11.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:34:11.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:11.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:34:11.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:34:11.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:34:11.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:34:11.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:11.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:34:11.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:34:11.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:34:11.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:34:11.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:34:11.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:34:11.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:34:11.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:34:11.482 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.482 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.482 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.482 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.483 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:11.483 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.686222964999999, 2024-04-24T14:34:11.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:34:11.483 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.483 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.483 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.483 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.483 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.483 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:34:11.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.483 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:34:11.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:34:11.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:34:11.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:34:11.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:34:11.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:34:11.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:34:11.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:34:11.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:34:11.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:11.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:34:11.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:34:11.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:34:11.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:34:11.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:11.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:34:11.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:34:11.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:11.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:34:11.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:34:11.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:34:11.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:34:11.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:34:11.489 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.489 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.489 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.489 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.489 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:11.490 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.686188389000002, 2024-04-24T14:34:11.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:34:11.490 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.490 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.490 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.490 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.490 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.490 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:34:11.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.490 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:11.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:11.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:11.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:11.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:34:11.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:34:11.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:34:11.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:34:11.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:34:11.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:34:11.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:34:11.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:34:11.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:11.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:34:11.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:34:11.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:34:11.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:34:11.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:11.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:34:11.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:34:11.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:34:11.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:34:11.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:34:11.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:34:11.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:34:11.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:34:11.496 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.496 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.496 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.497 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.497 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:11.497 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.686159050000001, 2024-04-24T14:34:11.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:34:11.497 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.497 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.497 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.497 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.497 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.497 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:34:11.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.497 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:11.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:11.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:34:11.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:34:11.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:34:11.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:34:11.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:34:11.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:34:11.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:34:11.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:34:11.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:34:11.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:11.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:34:11.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:34:11.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:34:11.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:34:11.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:11.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:34:11.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:34:11.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:34:11.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:34:11.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:34:11.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:34:11.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:34:11.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:34:11.503 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.503 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.503 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.503 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.503 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:11.504 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.686082467999999, 2024-04-24T14:34:11.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:34:11.504 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.504 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.504 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.504 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.504 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.504 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:34:11.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.504 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:11.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:11.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:34:11.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:34:11.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:34:11.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:34:11.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:34:11.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:34:11.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:34:11.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:34:11.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:34:11.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:11.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:34:11.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:34:11.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:34:11.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:34:11.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:11.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:34:11.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:34:11.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:34:11.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:34:11.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:34:11.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:34:11.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:34:11.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:34:11.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.510 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.510 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.510 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:11.510 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.686034923000001, 2024-04-24T14:34:11.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:34:11.511 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.511 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.511 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.511 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.511 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.511 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:34:11.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.511 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:11.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:11.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:34:11.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:34:11.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:34:11.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:34:11.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:34:11.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:34:11.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:34:11.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:34:11.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:34:11.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:11.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:34:11.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:34:11.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:34:11.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:34:11.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:11.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:34:11.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:34:11.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:34:11.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:34:11.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:34:11.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:34:11.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:34:11.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:34:11.517 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.517 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.517 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.517 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.517 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:11.518 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.685994280999999, 2024-04-24T14:34:11.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:34:11.518 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.518 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.518 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.518 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.518 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.518 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:34:11.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.518 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:11.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:11.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:11.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:11.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:34:11.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:34:11.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:34:11.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:34:11.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:34:11.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:34:11.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:34:11.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:34:11.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:11.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:34:11.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:34:11.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:34:11.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:34:11.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:11.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:34:11.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:34:11.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:34:11.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:34:11.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:34:11.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:34:11.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:34:11.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:34:11.524 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.524 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.524 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.524 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.525 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:11.525 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.685977170000001, 2024-04-24T14:34:11.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:34:11.525 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.525 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.525 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.525 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.525 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.525 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:34:11.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.525 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:11.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:11.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:34:11.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:11.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:34:11.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:34:11.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:34:11.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:34:11.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:34:11.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:34:11.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:34:11.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:34:11.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:11.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:34:11.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:34:11.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:34:11.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:34:11.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:11.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:34:11.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:34:11.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:34:11.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:34:11.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:34:11.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:34:11.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:34:11.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:34:11.531 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.531 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.531 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.531 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:481 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.531 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:11.531 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.652105463000002, 2024-04-24T14:34:11.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010661718000000001, 2024-04-24T14:34:11.532 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.532 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.532 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.532 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.532 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.532 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 481 2024-04-24T14:34:11.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.532 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:11.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:11.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793981+0000", 2024-04-24T14:34:11.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793985+0000", 2024-04-24T14:34:11.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8469999999999998e-06 2024-04-24T14:34:11.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:11.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0220000000000004e-06 2024-04-24T14:34:11.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793998+0000", 2024-04-24T14:34:11.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8559999999999998e-06 2024-04-24T14:34:11.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803807+0000", 2024-04-24T14:34:11.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098088629999999993 2024-04-24T14:34:11.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803916+0000", 2024-04-24T14:34:11.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010854899999999999 2024-04-24T14:34:11.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.804644+0000", 2024-04-24T14:34:11.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072835399999999996 2024-04-24T14:34:11.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.536 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.536 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:497 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.537 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:11.537 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.634165151000001, 2024-04-24T14:34:11.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013996471, 2024-04-24T14:34:11.537 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.537 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.537 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.537 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.537 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.537 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 497 2024-04-24T14:34:11.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.537 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:11.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:11.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811922+0000", 2024-04-24T14:34:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811925+0000", 2024-04-24T14:34:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4120000000000001e-06 2024-04-24T14:34:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811936+0000", 2024-04-24T14:34:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1077000000000001e-05 2024-04-24T14:34:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811941+0000", 2024-04-24T14:34:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7539999999999997e-06 2024-04-24T14:34:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822289+0000", 2024-04-24T14:34:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010347635000000001 2024-04-24T14:34:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822457+0000", 2024-04-24T14:34:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000168257 2024-04-24T14:34:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.825919+0000", 2024-04-24T14:34:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003461979 2024-04-24T14:34:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.542 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:25 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:11.542 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:11.542 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.47860721, 2024-04-24T14:34:11.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013349789000000001, 2024-04-24T14:34:11.542 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.542 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:11.542 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.542 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:11.542 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:11.542 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 25 2024-04-24T14:34:11.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.543 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:11.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:11.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967479+0000", 2024-04-24T14:34:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967669+0000", 2024-04-24T14:34:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000189829 2024-04-24T14:34:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967687+0000", 2024-04-24T14:34:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8199999999999999e-05 2024-04-24T14:34:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967695+0000", 2024-04-24T14:34:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4869999999999996e-06 2024-04-24T14:34:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.972501+0000", 2024-04-24T14:34:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048057509999999996 2024-04-24T14:34:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.972824+0000", 2024-04-24T14:34:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032322299999999998 2024-04-24T14:34:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.973071+0000", 2024-04-24T14:34:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024751299999999999 2024-04-24T14:34:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:11.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977624+0000", 2024-04-24T14:34:11.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045527570000000002 2024-04-24T14:34:11.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:11.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980627+0000", 2024-04-24T14:34:11.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003002972 2024-04-24T14:34:11.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:11.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980743+0000", 2024-04-24T14:34:11.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2024-04-24T14:34:11.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980830+0000", 2024-04-24T14:34:11.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7534999999999995e-05 2024-04-24T14:34:11.548 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.548 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.548 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:26 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.478383475000001, 2024-04-24T14:34:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013403597999999999, 2024-04-24T14:34:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 26 2024-04-24T14:34:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967703+0000", 2024-04-24T14:34:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967874+0000", 2024-04-24T14:34:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017088499999999999 2024-04-24T14:34:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967886+0000", 2024-04-24T14:34:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1744e-05 2024-04-24T14:34:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967893+0000", 2024-04-24T14:34:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2024-04-24T14:34:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974028+0000", 2024-04-24T14:34:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061356650000000002 2024-04-24T14:34:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974347+0000", 2024-04-24T14:34:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031897400000000003 2024-04-24T14:34:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974774+0000", 2024-04-24T14:34:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042661700000000001 2024-04-24T14:34:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980930+0000", 2024-04-24T14:34:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061563360000000001 2024-04-24T14:34:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980945+0000", 2024-04-24T14:34:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4864e-05 2024-04-24T14:34:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981035+0000", 2024-04-24T14:34:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9390999999999998e-05 2024-04-24T14:34:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981108+0000", 2024-04-24T14:34:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3255000000000003e-05 2024-04-24T14:34:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:27 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.478186329, 2024-04-24T14:34:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013394262000000001, 2024-04-24T14:34:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 27 2024-04-24T14:34:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967900+0000", 2024-04-24T14:34:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968061+0000", 2024-04-24T14:34:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016101900000000001 2024-04-24T14:34:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968072+0000", 2024-04-24T14:34:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1012999999999999e-05 2024-04-24T14:34:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968079+0000", 2024-04-24T14:34:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.46e-06 2024-04-24T14:34:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.976820+0000", 2024-04-24T14:34:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087406289999999998 2024-04-24T14:34:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.976951+0000", 2024-04-24T14:34:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131187 2024-04-24T14:34:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977129+0000", 2024-04-24T14:34:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000178135 2024-04-24T14:34:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981126+0000", 2024-04-24T14:34:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00399709 2024-04-24T14:34:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981193+0000", 2024-04-24T14:34:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7083999999999997e-05 2024-04-24T14:34:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981251+0000", 2024-04-24T14:34:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7754000000000003e-05 2024-04-24T14:34:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981296+0000", 2024-04-24T14:34:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4765000000000003e-05 2024-04-24T14:34:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.563 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:28 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:11.563 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:11.563 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.478001097, 2024-04-24T14:34:11.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013427247999999999, 2024-04-24T14:34:11.563 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.563 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:11.563 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.563 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:11.563 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:11.563 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 28 2024-04-24T14:34:11.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.564 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:11.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:11.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968086+0000", 2024-04-24T14:34:11.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968247+0000", 2024-04-24T14:34:11.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016121199999999999 2024-04-24T14:34:11.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968260+0000", 2024-04-24T14:34:11.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.275e-05 2024-04-24T14:34:11.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968266+0000", 2024-04-24T14:34:11.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3770000000000001e-06 2024-04-24T14:34:11.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977699+0000", 2024-04-24T14:34:11.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094331439999999992 2024-04-24T14:34:11.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977894+0000", 2024-04-24T14:34:11.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194945 2024-04-24T14:34:11.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:11.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.978258+0000", 2024-04-24T14:34:11.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036359400000000002 2024-04-24T14:34:11.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:11.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981140+0000", 2024-04-24T14:34:11.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028821540000000001 2024-04-24T14:34:11.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:11.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981360+0000", 2024-04-24T14:34:11.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022056300000000001 2024-04-24T14:34:11.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:11.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981444+0000", 2024-04-24T14:34:11.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3733999999999995e-05 2024-04-24T14:34:11.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981514+0000", 2024-04-24T14:34:11.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9702000000000005e-05 2024-04-24T14:34:11.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.569 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.570 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:29 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:11.570 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:11.570 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.477814252000002, 2024-04-24T14:34:11.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014492283, 2024-04-24T14:34:11.570 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.570 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:11.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:11.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:11.570 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 29 2024-04-24T14:34:11.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.570 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:11.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:11.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:11.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968434+0000", 2024-04-24T14:34:11.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016130199999999999 2024-04-24T14:34:11.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968484+0000", 2024-04-24T14:34:11.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0003000000000002e-05 2024-04-24T14:34:11.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968499+0000", 2024-04-24T14:34:11.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4628e-05 2024-04-24T14:34:11.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979299+0000", 2024-04-24T14:34:11.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010800624999999999 2024-04-24T14:34:11.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979488+0000", 2024-04-24T14:34:11.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000189037 2024-04-24T14:34:11.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:11.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979784+0000", 2024-04-24T14:34:11.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029562000000000002 2024-04-24T14:34:11.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:11.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982305+0000", 2024-04-24T14:34:11.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025212339999999998 2024-04-24T14:34:11.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:11.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982680+0000", 2024-04-24T14:34:11.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000374762 2024-04-24T14:34:11.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982731+0000", 2024-04-24T14:34:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1146999999999997e-05 2024-04-24T14:34:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982766+0000", 2024-04-24T14:34:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4755000000000002e-05 2024-04-24T14:34:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.576 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.577 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.577 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:11.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:11.579 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:11.583+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:11.679 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:11.685+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:11.728 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:11.729 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:11.729 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:11.729 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:11.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:11.780 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:11.783+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:11.839 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:11.840 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:11.840 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:11.840 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:11.840 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:11.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:11.881 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:11.885+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T14:34:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-24T14:34:11.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T14:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:192 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.494774079999999, 2024-04-24T14:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010036329, 2024-04-24T14:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.961 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192 2024-04-24T14:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471099+0000", 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471103+0000", 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8800000000000001e-06 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471111+0000", 2024-04-24T14:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2930000000000001e-06 2024-04-24T14:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471116+0000", 2024-04-24T14:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1800000000000004e-06 2024-04-24T14:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480338+0000", 2024-04-24T14:34:11.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092213069999999998 2024-04-24T14:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480467+0000", 2024-04-24T14:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012901500000000001 2024-04-24T14:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481136+0000", 2024-04-24T14:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066948400000000003 2024-04-24T14:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.494751044000001, 2024-04-24T14:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.966 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:34:11.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.119394341, 2024-04-24T14:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.970 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:526 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.974 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.118006818000001, 2024-04-24T14:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010568783999999999, 2024-04-24T14:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 526 2024-04-24T14:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847866+0000", 2024-04-24T14:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847871+0000", 2024-04-24T14:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1590000000000001e-06 2024-04-24T14:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847883+0000", 2024-04-24T14:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.172e-05 2024-04-24T14:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847890+0000", 2024-04-24T14:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6950000000000003e-06 2024-04-24T14:34:11.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857418+0000", 2024-04-24T14:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095285909999999995 2024-04-24T14:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857588+0000", 2024-04-24T14:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169541 2024-04-24T14:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858436+0000", 2024-04-24T14:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084814700000000005 2024-04-24T14:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.117966187, 2024-04-24T14:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:11.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.983 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.117939098000001, 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.117915131, 2024-04-24T14:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.988 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:11.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.992 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.103931152000001, 2024-04-24T14:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:34:11.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:34:11.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:11.998 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.103814840999998, 2024-04-24T14:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:11.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:12.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.103767182999999, 2024-04-24T14:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:34:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:34:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:34:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.099729884999999, 2024-04-24T14:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:34:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:12.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:34:12.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:34:12.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:34:12.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:34:12.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:34:12.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:34:12.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:34:12.016 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.016 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:12.016 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.016 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:12.016 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:12.016 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.099666125999999, 2024-04-24T14:34:12.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:34:12.016 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:12.016 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:12.016 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:12.016 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:12.016 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.017 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:34:12.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.017 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:12.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:12.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:12.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:12.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:12.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:12.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:34:12.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:12.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:34:12.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:12.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:12.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:34:12.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:34:12.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:34:12.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:34:12.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:34:12.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:34:12.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:34:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:34:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:34:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:34:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:34:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:34:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:34:12.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:34:12.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:12.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:34:12.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:34:12.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:12.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:34:12.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:34:12.021 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.021 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:12.022 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.022 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:12.022 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:12.022 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.099640636, 2024-04-24T14:34:12.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:34:12.022 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:12.022 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:12.022 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:12.022 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:12.022 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.022 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:34:12.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:12.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:34:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:34:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:34:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:34:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:34:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:34:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:34:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:34:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:34:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:34:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:34:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:34:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:34:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:34:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:34:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:34:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:34:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:34:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:34:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.099616886, 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:34:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:34:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:34:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:34:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:34:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:34:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:34:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:34:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:34:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:34:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:34:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:34:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:34:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:34:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:34:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:34:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.099578373, 2024-04-24T14:34:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:34:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:34:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:34:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:34:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:34:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:34:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:34:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:34:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:34:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:34:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:34:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:34:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:34:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:34:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:34:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:34:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:34:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.099550176000001, 2024-04-24T14:34:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:34:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:34:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:34:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:34:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:34:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:34:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:34:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:34:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:34:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:34:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:34:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:34:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:34:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:34:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:34:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.099528718999998, 2024-04-24T14:34:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:34:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:34:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:34:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:34:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:34:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:34:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:34:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:34:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:34:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:34:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:34:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:34:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:34:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:34:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:34:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:34:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:34:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.099509876999999, 2024-04-24T14:34:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:34:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:34:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:34:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:34:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:34:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:34:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:34:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:34:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:34:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:34:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:34:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:34:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:34:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:34:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:34:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:34:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:34:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:561 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.057547257, 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01066806, 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 561 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908326+0000", 2024-04-24T14:34:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908329+0000", 2024-04-24T14:34:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.802e-06 2024-04-24T14:34:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908338+0000", 2024-04-24T14:34:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6260000000000001e-06 2024-04-24T14:34:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908345+0000", 2024-04-24T14:34:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1389999999999998e-06 2024-04-24T14:34:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.917978+0000", 2024-04-24T14:34:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096329840000000007 2024-04-24T14:34:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918178+0000", 2024-04-24T14:34:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019999199999999999 2024-04-24T14:34:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918995+0000", 2024-04-24T14:34:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081708099999999999 2024-04-24T14:34:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 12.886677728, 2024-04-24T14:34:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:34:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:34:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:34:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:34:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:34:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:34:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:34:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:34:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:34:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:34:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:34:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:34:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:34:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:34:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:34:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:34:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:34:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:34:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:34:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:34:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:34:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:34:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:12.067 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:12.068 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:12.069 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:12.071+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:12.169 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:12.173+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:12.174 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:12.174 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:12.174 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:12.174 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:12.184 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:12.270 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:12.273+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:12.280 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:12.280 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:12.280 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:12.280 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:12.280 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:12.289 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:12.370 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:12.373+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:12.395 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:12.395 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:12.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:12.395 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:12.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.395 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:12.395 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:12.395 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 109.771865299, 2024-04-24T14:34:12.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:34:12.395 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.395 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:12.395 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.396 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:12.396 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:12.396 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:34:12.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.396 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:12.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:12.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:34:12.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:12.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:34:12.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:34:12.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:34:12.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:34:12.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:34:12.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:34:12.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:34:12.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:34:12.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:34:12.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:34:12.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:12.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:34:12.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:34:12.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:12.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:34:12.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:34:12.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:12.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:34:12.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:34:12.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:12.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:34:12.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:34:12.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:34:12.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:34:12.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.401 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.401 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:12.401 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:12.401 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 109.771688212, 2024-04-24T14:34:12.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:34:12.401 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.401 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:12.401 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.401 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:12.402 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:12.402 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:34:12.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.402 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:12.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:12.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:34:12.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:12.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:34:12.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:34:12.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:34:12.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:34:12.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:34:12.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:34:12.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:34:12.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:34:12.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:34:12.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:34:12.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:12.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:34:12.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:34:12.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:12.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:34:12.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:34:12.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:34:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:34:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:34:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:34:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:34:12.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:34:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 109.77158429799999, 2024-04-24T14:34:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:34:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:12.407 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:34:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:34:12.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:12.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:34:12.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:34:12.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:34:12.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:34:12.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:34:12.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:12.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:34:12.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:34:12.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:12.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:34:12.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:34:12.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:12.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:34:12.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:34:12.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:34:12.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:34:12.412 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.413 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.413 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.413 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:12.413 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:12.413 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 109.771195847, 2024-04-24T14:34:12.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:34:12.413 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.413 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:12.413 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.413 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:12.413 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:12.414 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:34:12.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.414 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:12.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:12.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:34:12.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:12.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:34:12.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:34:12.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:34:12.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:34:12.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:34:12.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:34:12.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:34:12.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:34:12.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:34:12.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:34:12.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:12.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:34:12.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:34:12.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:12.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:34:12.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:34:12.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:12.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:34:12.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:34:12.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:12.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:34:12.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:34:12.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:34:12.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:34:12.418 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.418 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.419 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.419 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:12.419 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:12.419 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 109.771110635, 2024-04-24T14:34:12.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:34:12.419 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.419 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:12.419 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.419 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:12.419 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:12.419 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:12.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.420 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:12.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:12.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:34:12.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:34:12.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:34:12.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:34:12.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:34:12.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:34:12.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:12.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:34:12.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:34:12.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:12.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:34:12.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:34:12.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:12.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:34:12.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:34:12.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:12.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:34:12.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:34:12.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:34:12.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:34:12.424 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.424 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.424 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 109.77087375399999, 2024-04-24T14:34:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:34:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:34:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.425 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:12.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:12.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:34:12.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:12.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:34:12.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:34:12.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:34:12.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:34:12.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:34:12.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:34:12.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:34:12.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:34:12.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:34:12.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:34:12.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:12.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:34:12.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:34:12.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:12.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:34:12.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:34:12.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:12.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:34:12.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:34:12.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:34:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:34:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:34:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:34:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.431 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:12.431 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:12.431 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 109.77079223200001, 2024-04-24T14:34:12.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:34:12.431 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.431 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:12.431 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.431 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:12.431 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:12.431 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:34:12.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.431 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:34:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:34:12.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:34:12.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:34:12.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:34:12.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:34:12.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:34:12.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:34:12.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:34:12.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:34:12.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:34:12.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:12.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:34:12.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:34:12.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:12.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:34:12.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:34:12.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:12.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:34:12.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:34:12.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:12.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:34:12.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:34:12.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:34:12.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:34:12.436 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.436 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.436 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.436 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:12.436 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:12.437 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 109.338879885, 2024-04-24T14:34:12.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:34:12.437 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.437 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:12.437 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.437 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:12.437 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:12.437 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:34:12.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.437 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:12.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:12.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:34:12.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:34:12.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:34:12.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:34:12.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:34:12.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:34:12.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:34:12.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:34:12.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:34:12.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:34:12.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:34:12.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:12.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:34:12.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:34:12.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:12.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:34:12.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:34:12.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:12.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:34:12.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:34:12.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:12.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:34:12.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:34:12.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:34:12.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:34:12.442 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.442 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.442 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.442 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:12.442 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:12.443 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 109.338851585, 2024-04-24T14:34:12.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:34:12.443 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.443 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:12.443 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.443 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:12.443 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:12.443 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:34:12.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.443 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:12.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:12.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:34:12.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:34:12.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:34:12.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:34:12.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:34:12.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:34:12.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:34:12.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:34:12.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:34:12.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:34:12.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:34:12.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:12.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:34:12.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:34:12.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:12.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:34:12.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:34:12.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:12.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:34:12.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:34:12.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:12.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:34:12.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:34:12.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:34:12.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:34:12.448 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.448 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.448 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.448 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:12.448 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:12.448 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 109.33876388100001, 2024-04-24T14:34:12.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:34:12.449 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.449 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:12.449 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.449 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:12.449 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:12.449 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:34:12.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.449 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:12.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:12.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:34:12.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:34:12.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:34:12.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:34:12.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:34:12.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:34:12.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:34:12.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:34:12.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:34:12.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:34:12.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:34:12.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:12.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:34:12.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:34:12.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:12.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:34:12.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:34:12.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:12.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:34:12.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:34:12.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:12.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:34:12.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:34:12.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:34:12.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:34:12.454 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.454 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.454 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.454 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:12.454 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:12.454 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 109.33864108, 2024-04-24T14:34:12.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:34:12.454 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.455 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:12.455 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.455 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:12.455 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:12.455 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:34:12.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.455 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:12.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:12.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:34:12.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:34:12.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:34:12.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:34:12.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:34:12.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:34:12.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:34:12.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:34:12.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:34:12.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:34:12.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:34:12.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:12.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:34:12.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:34:12.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:12.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:34:12.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:34:12.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:12.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:34:12.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:34:12.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:12.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:34:12.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:34:12.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:34:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:34:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:34:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 40.032396519999999, 2024-04-24T14:34:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:34:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:12.460 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.461 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:34:12.461 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:34:12.461 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:34:12.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.461 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:12.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:12.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:34:12.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:12.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:34:12.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:34:12.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:34:12.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:34:12.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:34:12.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:34:12.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:34:12.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:34:12.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:34:12.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:34:12.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:34:12.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:34:12.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:34:12.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:12.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:34:12.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:34:12.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:34:12.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:34:12.465 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.465 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.465 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.465 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.465 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:12.465 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.338272521, 2024-04-24T14:34:12.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:34:12.466 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.466 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.466 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.466 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:34:12.466 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:34:12.466 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:34:12.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.466 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:12.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:12.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:34:12.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:12.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:34:12.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:34:12.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:34:12.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:34:12.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:34:12.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:34:12.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:34:12.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:34:12.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:34:12.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:34:12.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:34:12.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:34:12.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:34:12.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:34:12.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:34:12.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:34:12.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:34:12.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:34:12.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:34:12.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:34:12.471 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.471 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.471 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.471 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:12.471 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:12.471 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.378912554999999, 2024-04-24T14:34:12.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:34:12.471 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.472 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.472 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.472 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:12.472 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:12.472 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:34:12.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.472 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:12.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:12.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:34:12.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:34:12.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:34:12.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:34:12.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:34:12.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:34:12.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:34:12.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:34:12.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:34:12.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:34:12.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:34:12.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:34:12.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:34:12.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:34:12.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:34:12.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:34:12.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:34:12.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:34:12.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:34:12.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:34:12.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:34:12.477 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.477 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.477 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.477 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:12.477 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:12.477 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.378831999999999, 2024-04-24T14:34:12.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:34:12.477 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.477 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:12.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:12.478 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:34:12.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.478 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:12.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:12.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:34:12.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:34:12.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:34:12.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:34:12.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:34:12.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:34:12.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:34:12.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:34:12.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:34:12.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:34:12.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:34:12.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:34:12.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:34:12.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:34:12.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:34:12.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:34:12.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:34:12.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:34:12.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:34:12.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:34:12.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:34:12.483 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.483 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.483 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.483 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:12.483 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:12.483 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.378784271000001, 2024-04-24T14:34:12.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:34:12.484 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.484 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.484 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.484 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:12.484 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:12.484 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:34:12.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.484 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:12.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:12.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:34:12.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:34:12.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:34:12.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:34:12.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:34:12.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:34:12.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:34:12.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:34:12.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:34:12.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:34:12.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:34:12.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:34:12.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:34:12.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:34:12.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:34:12.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:34:12.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:34:12.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:34:12.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:34:12.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:34:12.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:34:12.489 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.489 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.489 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.489 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:12.489 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:12.490 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.378756508, 2024-04-24T14:34:12.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:34:12.490 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.490 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.490 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.490 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:12.490 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:12.490 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:34:12.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.490 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:12.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:12.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:34:12.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:34:12.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:34:12.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:34:12.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:34:12.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:34:12.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:34:12.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:34:12.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:34:12.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:34:12.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:34:12.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:34:12.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:34:12.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:34:12.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:34:12.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:34:12.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:34:12.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:34:12.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:34:12.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:34:12.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:34:12.495 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.495 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.495 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.496 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:12.496 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:12.496 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.378717073000001, 2024-04-24T14:34:12.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:34:12.496 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.496 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.496 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.496 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:12.496 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:12.496 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:34:12.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.496 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:12.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:12.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:34:12.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:34:12.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:34:12.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:34:12.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:34:12.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:34:12.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:34:12.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:34:12.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:34:12.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:34:12.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:34:12.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:34:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:34:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:34:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:34:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:34:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:34:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:34:12.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:34:12.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:34:12.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:34:12.501 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.501 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.501 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.501 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:12.502 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:12.502 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.378692088999999, 2024-04-24T14:34:12.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:34:12.502 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.502 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.502 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.502 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:12.502 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:12.502 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:34:12.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.502 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:12.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:12.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:34:12.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:34:12.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:34:12.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:34:12.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:34:12.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:34:12.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:34:12.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:34:12.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:34:12.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:34:12.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:34:12.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:34:12.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:34:12.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:34:12.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:34:12.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:34:12.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:34:12.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:34:12.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:34:12.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:34:12.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:34:12.507 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.507 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.507 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.507 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:12.508 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:12.508 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 13.378666429999999, 2024-04-24T14:34:12.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:34:12.508 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.508 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.508 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.508 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:12.508 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:12.508 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:34:12.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.508 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:34:12.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:34:12.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:34:12.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:34:12.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:34:12.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:12.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:34:12.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:34:12.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:12.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:34:12.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:34:12.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:34:12.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:34:12.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:12.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:34:12.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:34:12.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:34:12.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:34:12.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:34:12.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:34:12.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:34:12.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:34:12.513 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.513 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.513 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.513 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.513 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.513 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:12.515 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:12.515 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:12.521+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:12.618 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:12.621+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:12.624 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:12.624 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:12.624 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:12.624 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:12.633 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:12.719 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:12.725+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:12.780 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:12.780 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:12.780 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:12.780 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:12.780 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:12.789 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:12.821 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:12.823+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:12.894 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:12.894 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:12.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:12.895 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:12.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.895 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:379 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.895 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:12.895 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.234046857999999, 2024-04-24T14:34:12.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068078410000000002, 2024-04-24T14:34:12.895 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.895 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.895 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.895 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.895 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.896 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 379 2024-04-24T14:34:12.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.896 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:12.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:12.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:12.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664240+0000", 2024-04-24T14:34:12.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6539999999999999e-06 2024-04-24T14:34:12.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.898 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664245+0000", 2024-04-24T14:34:12.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2179999999999998e-06 2024-04-24T14:34:12.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.898 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.898 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664249+0000", 2024-04-24T14:34:12.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2279999999999998e-06 2024-04-24T14:34:12.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.898 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.898 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670245+0000", 2024-04-24T14:34:12.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059962430000000001 2024-04-24T14:34:12.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670400+0000", 2024-04-24T14:34:12.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015535399999999999 2024-04-24T14:34:12.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671046+0000", 2024-04-24T14:34:12.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064591599999999996 2024-04-24T14:34:12.899 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.899 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.899 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.900 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:380 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.900 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:34:12.900 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.234000231, 2024-04-24T14:34:12.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076160830000000001, 2024-04-24T14:34:12.900 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.900 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.900 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.900 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.900 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.900 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 380 2024-04-24T14:34:12.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.901 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:34:12.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:34:12.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664284+0000", 2024-04-24T14:34:12.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664287+0000", 2024-04-24T14:34:12.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9739999999999998e-06 2024-04-24T14:34:12.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664302+0000", 2024-04-24T14:34:12.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5214999999999999e-05 2024-04-24T14:34:12.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664307+0000", 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6859999999999997e-06 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671098+0000", 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067912479999999997 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671253+0000", 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015427400000000001 2024-04-24T14:34:12.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671901+0000", 2024-04-24T14:34:12.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064845600000000005 2024-04-24T14:34:12.904 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.904 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.904 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.904 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.904 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:12.904 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.233970685999999, 2024-04-24T14:34:12.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:34:12.904 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.905 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.905 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.905 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.905 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.905 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:34:12.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.905 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:12.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:12.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:12.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:34:12.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:34:12.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:34:12.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:34:12.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:34:12.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:34:12.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:34:12.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:34:12.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.908 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:382 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.909 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:34:12.909 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.230969203000001, 2024-04-24T14:34:12.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067029910000000002, 2024-04-24T14:34:12.909 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.909 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.909 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.909 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.909 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.909 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 382 2024-04-24T14:34:12.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.909 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:34:12.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:34:12.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667315+0000", 2024-04-24T14:34:12.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667321+0000", 2024-04-24T14:34:12.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2099999999999998e-06 2024-04-24T14:34:12.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667336+0000", 2024-04-24T14:34:12.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4612e-05 2024-04-24T14:34:12.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667343+0000", 2024-04-24T14:34:12.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0119999999999999e-06 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672957+0000", 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056139700000000002 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.673127+0000", 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017044599999999999 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674019+0000", 2024-04-24T14:34:12.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00089165499999999999 2024-04-24T14:34:12.913 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.913 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.913 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.913 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.913 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:12.913 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.230936159999999, 2024-04-24T14:34:12.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:34:12.913 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.913 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.913 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.914 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.914 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.914 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:34:12.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.914 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:12.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:34:12.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:34:12.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:34:12.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:34:12.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:34:12.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:34:12.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:12.917 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.230911974000001, 2024-04-24T14:34:12.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:34:12.918 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.918 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.918 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.918 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.918 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.918 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:34:12.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.918 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:12.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:12.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:12.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:34:12.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:34:12.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:34:12.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:34:12.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:34:12.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:34:12.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:34:12.921 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.921 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.921 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.922 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.922 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:12.922 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.230894589999998, 2024-04-24T14:34:12.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:34:12.922 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.922 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.922 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.922 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.922 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.922 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:34:12.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.922 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:34:12.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:34:12.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:34:12.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:34:12.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:34:12.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:34:12.926 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.926 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.926 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.926 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:399 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.926 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:34:12.926 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.216484455, 2024-04-24T14:34:12.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071317810000000002, 2024-04-24T14:34:12.927 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.927 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.927 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.927 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.927 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.927 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 399 2024-04-24T14:34:12.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.927 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:34:12.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:34:12.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681800+0000", 2024-04-24T14:34:12.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681804+0000", 2024-04-24T14:34:12.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.715e-06 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681816+0000", 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1851000000000001e-05 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681820+0000", 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6450000000000004e-06 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688059+0000", 2024-04-24T14:34:12.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062390090000000002 2024-04-24T14:34:12.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688196+0000", 2024-04-24T14:34:12.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013721999999999999 2024-04-24T14:34:12.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688933+0000", 2024-04-24T14:34:12.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073605000000000003 2024-04-24T14:34:12.930 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.930 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.930 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.216458498000001, 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:34:12.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:34:12.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:34:12.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:34:12.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:34:12.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:34:12.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:34:12.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:34:12.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:34:12.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:34:12.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:34:12.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:34:12.934 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.216426199000001, 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:34:12.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.936 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:12.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:12.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:34:12.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:12.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:34:12.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:34:12.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:34:12.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:34:12.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:34:12.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:34:12.939 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.939 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.939 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.939 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:414 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.939 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:34:12.939 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.202230789000001, 2024-04-24T14:34:12.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069877910000000001, 2024-04-24T14:34:12.939 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.939 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.940 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.940 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.940 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.940 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 414 2024-04-24T14:34:12.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.940 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:34:12.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696053+0000", 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696058+0000", 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7509999999999999e-06 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696074+0000", 2024-04-24T14:34:12.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5743000000000002e-05 2024-04-24T14:34:12.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696093+0000", 2024-04-24T14:34:12.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9508e-05 2024-04-24T14:34:12.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702314+0000", 2024-04-24T14:34:12.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062207010000000004 2024-04-24T14:34:12.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702431+0000", 2024-04-24T14:34:12.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011723299999999999 2024-04-24T14:34:12.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703042+0000", 2024-04-24T14:34:12.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000610752 2024-04-24T14:34:12.943 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.943 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.943 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.943 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.943 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.202173922, 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:12.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:34:12.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:34:12.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:34:12.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.948 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.948 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:12.948 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.202144822000001, 2024-04-24T14:34:12.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:34:12.948 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.948 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.948 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.948 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.948 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.948 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:34:12.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.948 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:34:12.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:34:12.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:34:12.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:34:12.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:34:12.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:34:12.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.951 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.202118892000001, 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.952 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:12.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:12.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:12.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:34:12.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:34:12.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:34:12.956 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.956 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.956 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.956 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:429 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3538944~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.956 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:34:12.956 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.187837452, 2024-04-24T14:34:12.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071866789999999996, 2024-04-24T14:34:12.956 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.956 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.956 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.956 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.956 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 429 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710447+0000", 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710451+0000", 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5930000000000002e-06 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710465+0000", 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.411e-05 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710470+0000", 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3129999999999998e-06 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716696+0000", 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006225983 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716901+0000", 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020550200000000001 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717634+0000", 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073292099999999998 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:430 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.186473144000001, 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068326970000000004, 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.960 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 430 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711810+0000", 2024-04-24T14:34:12.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:12.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711822+0000", 2024-04-24T14:34:12.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2166e-05 2024-04-24T14:34:12.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711856+0000", 2024-04-24T14:34:12.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3816000000000003e-05 2024-04-24T14:34:12.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711873+0000", 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6742999999999999e-05 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717693+0000", 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058204579999999997 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717864+0000", 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017128299999999999 2024-04-24T14:34:12.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718645+0000", 2024-04-24T14:34:12.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078033699999999998 2024-04-24T14:34:12.964 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.964 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.964 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.964 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.964 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:12.964 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.186398428, 2024-04-24T14:34:12.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:34:12.964 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.965 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:12.965 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.965 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.965 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.965 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:34:12.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.965 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:12.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:34:12.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:34:12.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:34:12.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:34:12.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:34:12.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:34:12.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.968 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.186366496000002, 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:12.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:34:12.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:12.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:34:12.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:34:12.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.18629898, 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:34:12.972 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:12.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:34:12.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:12.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:34:12.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:34:12.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.128886096999999, 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:12.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:12.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:12.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:12.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:12.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:12.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:12.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:12.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:12.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:12.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:34:12.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:34:12.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:12.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:34:12.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:34:12.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:12.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:34:12.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:34:12.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:12.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:12.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:12.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:34:12.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:34:12.980 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.980 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.980 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.980 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:12.980 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:12.980 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:12.982 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T14:34:12.982 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:12.983 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:12.985+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:12.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T14:34:13.082 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:13.085+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:13.131 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:13.131 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:13.131 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:13.131 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:13.140 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:13.183 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:13.189+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:13.237 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:13.238 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:13.238 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:13.238 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:13.238 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:13.246 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:13.291 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:13.297+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:13.384 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:13.391+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:13.396 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:13.396 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:13.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:13.396 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:13.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.397 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:34:13.397 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:13.397 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 33.328405019999998, 2024-04-24T14:34:13.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:34:13.397 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.397 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.397 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:13.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:13.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:34:13.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:13.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:34:13.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:34:13.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:34:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:34:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:34:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:34:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:34:13.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:34:13.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:34:13.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:34:13.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:34:13.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:34:13.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:34:13.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:34:13.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.401 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.401 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:34:13.401 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:13.401 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.333590440999998, 2024-04-24T14:34:13.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:34:13.401 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.401 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.401 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:13.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:13.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:34:13.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:13.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:34:13.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:34:13.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:34:13.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:34:13.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:34:13.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:34:13.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:34:13.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:34:13.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:34:13.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:34:13.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:34:13.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:34:13.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:34:13.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:34:13.405 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.405 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.405 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.405 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:34:13.406 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:13.406 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.322661183000001, 2024-04-24T14:34:13.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:34:13.406 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.406 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.406 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:13.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:13.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:34:13.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:13.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:34:13.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:34:13.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.409 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.410 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.410 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:34:13.410 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:13.410 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.311705102000001, 2024-04-24T14:34:13.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:34:13.410 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.410 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.410 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:34:13.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:34:13.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:34:13.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:34:13.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:34:13.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:34:13.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:34:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:34:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:34:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:34:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:34:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:34:13.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:34:13.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:34:13.414 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.414 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.414 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.414 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:34:13.414 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:13.414 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.300135737000002, 2024-04-24T14:34:13.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:34:13.414 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.415 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.415 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:13.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:13.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:34:13.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:13.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:34:13.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:34:13.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:34:13.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:34:13.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:34:13.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:34:13.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:34:13.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:34:13.418 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.418 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.418 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:109 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'60, mlcod=62'58)", 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.057023571999999, 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036290020000000001, 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343313+0000", 2024-04-24T14:34:13.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343311+0000", 2024-04-24T14:34:13.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:13.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343841+0000", 2024-04-24T14:34:13.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053003399999999997 2024-04-24T14:34:13.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343876+0000", 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4479e-05 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343888+0000", 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1623e-05 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.343996+0000", 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108028 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.344044+0000", 2024-04-24T14:34:13.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8081999999999998e-05 2024-04-24T14:34:13.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.346872+0000", 2024-04-24T14:34:13.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028280789999999998 2024-04-24T14:34:13.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.346942+0000", 2024-04-24T14:34:13.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0405000000000001e-05 2024-04-24T14:34:13.422 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.422 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.047016077999999, 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:13.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:34:13.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:13.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:34:13.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:34:13.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:34:13.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:34:13.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:34:13.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:34:13.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.036270909999999, 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.427 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:13.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:34:13.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:34:13.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:34:13.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:34:13.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:34:13.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:34:13.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:34:13.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:34:13.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:34:13.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:34:13.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:34:13.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:34:13.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:34:13.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:34:13.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.431 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.431 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:34:13.431 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:13.431 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.023904634000001, 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:34:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:34:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:34:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:34:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:34:13.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:34:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:34:13.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:34:13.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:34:13.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:34:13.435 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.435 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.435 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:34:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.898968513, 2024-04-24T14:34:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:34:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:34:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:34:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:34:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:34:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:34:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:34:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:34:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:138 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'70, mlcod=62'68)", 2024-04-24T14:34:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:34:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.888582359000001, 2024-04-24T14:34:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0036613330000000001, 2024-04-24T14:34:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511754+0000", 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.511753+0000", 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512370+0000", 2024-04-24T14:34:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061776500000000005 2024-04-24T14:34:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512395+0000", 2024-04-24T14:34:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4549000000000002e-05 2024-04-24T14:34:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512404+0000", 2024-04-24T14:34:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.1130000000000007e-06 2024-04-24T14:34:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512517+0000", 2024-04-24T14:34:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112679 2024-04-24T14:34:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.512581+0000", 2024-04-24T14:34:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4274000000000002e-05 2024-04-24T14:34:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.515303+0000", 2024-04-24T14:34:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027220640000000002 2024-04-24T14:34:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.515416+0000", 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112644 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.879580669999999, 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:34:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:34:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:34:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:34:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:34:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:34:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:34:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:34:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:34:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:34:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:34:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:34:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:34:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:34:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:34:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:34:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.869852826999999, 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:34:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:34:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:34:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:34:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:34:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:34:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:34:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:34:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:34:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:34:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:34:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:34:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:34:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:34:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:34:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:172 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'76, mlcod=62'74)", 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.719585472999999, 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0032080400000000001, 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680751+0000", 2024-04-24T14:34:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.680750+0000", 2024-04-24T14:34:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681277+0000", 2024-04-24T14:34:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052776400000000001 2024-04-24T14:34:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681304+0000", 2024-04-24T14:34:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6948e-05 2024-04-24T14:34:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681315+0000", 2024-04-24T14:34:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0648e-05 2024-04-24T14:34:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681413+0000", 2024-04-24T14:34:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7926999999999999e-05 2024-04-24T14:34:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.681470+0000", 2024-04-24T14:34:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7401999999999997e-05 2024-04-24T14:34:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.683923+0000", 2024-04-24T14:34:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024526790000000001 2024-04-24T14:34:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.683959+0000", 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6327e-05 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.709743147000001, 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:34:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:34:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:34:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:34:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:34:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:34:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:34:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:34:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:34:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:34:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:34:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.700014151000001, 2024-04-24T14:34:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:34:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:34:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:34:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:34:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:34:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:34:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:34:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:34:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:34:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:34:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:34:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:34:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:34:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:34:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:34:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:34:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:178 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'82, mlcod=62'80)", 2024-04-24T14:34:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:34:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.689546775, 2024-04-24T14:34:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029527640000000001, 2024-04-24T14:34:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:34:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710790+0000", 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.710789+0000", 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711538+0000", 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00074893099999999999 2024-04-24T14:34:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711565+0000", 2024-04-24T14:34:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7526999999999999e-05 2024-04-24T14:34:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711576+0000", 2024-04-24T14:34:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0553999999999999e-05 2024-04-24T14:34:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711633+0000", 2024-04-24T14:34:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7258e-05 2024-04-24T14:34:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.711653+0000", 2024-04-24T14:34:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0460999999999999e-05 2024-04-24T14:34:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.713712+0000", 2024-04-24T14:34:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020587790000000002 2024-04-24T14:34:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.713743+0000", 2024-04-24T14:34:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0519999999999999e-05 2024-04-24T14:34:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:34:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 14.365043316, 2024-04-24T14:34:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:34:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:34:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:34:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:34:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:34:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:34:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:34:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:34:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:34:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:34:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:34:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.520892568000001, 2024-04-24T14:34:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:34:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:34:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:34:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:34:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:34:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:34:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:34:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:34:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.957086+0000", 2024-04-24T14:34:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 10.443251223000001, 2024-04-24T14:34:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0029645230000000002, 2024-04-24T14:34:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957086+0000", 2024-04-24T14:34:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957086+0000", 2024-04-24T14:34:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957084+0000", 2024-04-24T14:34:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957106+0000", 2024-04-24T14:34:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1866e-05 2024-04-24T14:34:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957130+0000", 2024-04-24T14:34:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4266000000000001e-05 2024-04-24T14:34:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957140+0000", 2024-04-24T14:34:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.007e-05 2024-04-24T14:34:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.957264+0000", 2024-04-24T14:34:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012356699999999999 2024-04-24T14:34:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.960050+0000", 2024-04-24T14:34:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027863570000000002 2024-04-24T14:34:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:13.482 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:13.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:34:13.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:13.485 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:13.489+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:13.585 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:13.589+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:13.593 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:13.593 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:13.594 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:13.594 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:13.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:13.624 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:13.646 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:13.666 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:13.686 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:13.689+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:13.686 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:13.696 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:13.696 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:13.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:13.709 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:13.732 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:13.734 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:13.734 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:13.734 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:13.734 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:13.735 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:13.735 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:13.787 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:13.793+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:13.802 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:13.802 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:13.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:13.888 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:13.893+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:13.910 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:13.910 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:13.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:13.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T14:34:13.961 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T14:34:13.972 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T14:34:13.988 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:13.991+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:14.050 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:14.050 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:14.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:14.102 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:14.107+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:14.191 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:14.193+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:14.212 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:14.212 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:14.221 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:14.291 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:14.295+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:14.312 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:14.312 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:14.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:14.392 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:14.395+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:14.467 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:14.467 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:14.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:14.493 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:14.499+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:14.578 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:14.578 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:14.587 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:14.631 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:14.637+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:14.695 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:14.699+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:14.699 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:14.700 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:14.709 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:14.796 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:14.801+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:14.810 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:14.810 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:14.820 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:14.896 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:14.901+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:14.920 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:14.920 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:14.929 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:14.965 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T14:34:14.968 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-24T14:34:14.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-24T14:34:14.997 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:15.003+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:15.074 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:15.074 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:15.083 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:15.098 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:15.105+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:15.179 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:15.179 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:15.188 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:15.231 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:15.237+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:15.300 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:15.303+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:15.333 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:15.333 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:15.342 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:15.401 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:15.405+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:15.445 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:15.445 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:15.455 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:15.501 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:15.507+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:15.600 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:15.600 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:15.602 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:15.605+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:15.609 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:15.704 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:15.707+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:15.711 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:15.711 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:15.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:15.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:34:15.804 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:15.809+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:15.831 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:15.831 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:15.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:15.906 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:15.911+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:15.907 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:15.929 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:15.932 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:15.932 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:15.933 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:15.933 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:15.933 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:15.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:15.952 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:15.978 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.4 to 1 (10000) 2024-04-24T14:34:15.997 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T14:34:15.997 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:34:16.000 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:16.007 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:16.013+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:16.022 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:16.023 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:16.024 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:16.056 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.179707707, 2024-04-24T14:34:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:34:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:34:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:16.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:34:16.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:16.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:34:16.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:34:16.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:34:16.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:34:16.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:34:16.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:34:16.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:34:16.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:34:16.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:16.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:34:16.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:34:16.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:34:16.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:34:16.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:16.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:34:16.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:34:16.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:34:16.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:34:16.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:34:16.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:34:16.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:34:16.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:34:16.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.062 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.062 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:16.062 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:16.062 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.179678722, 2024-04-24T14:34:16.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:34:16.063 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.063 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:16.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:16.063 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:34:16.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.063 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:16.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:16.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:16.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:34:16.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:34:16.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:16.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:34:16.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:34:16.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:34:16.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:34:16.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:34:16.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:34:16.068 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.068 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.068 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.068 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:16.068 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:16.068 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.179660237, 2024-04-24T14:34:16.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:34:16.068 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.068 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:16.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:34:16.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:34:16.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:34:16.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:16.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:34:16.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:34:16.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:34:16.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:34:16.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:16.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:34:16.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:34:16.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:34:16.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:34:16.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:34:16.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:34:16.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:34:16.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:34:16.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.074 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.074 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:16.074 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:16.074 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.179644956000001, 2024-04-24T14:34:16.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:34:16.074 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.074 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:16.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:16.075 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:34:16.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.075 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:16.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:16.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:16.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:34:16.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:34:16.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:34:16.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:34:16.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:34:16.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:34:16.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:34:16.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:34:16.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:16.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:34:16.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:34:16.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:34:16.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:34:16.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:16.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:34:16.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:34:16.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:34:16.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.080 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.080 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:16.080 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:16.080 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.179629175000001, 2024-04-24T14:34:16.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:34:16.080 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.080 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:16.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:16.081 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:34:16.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.081 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:16.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:16.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:16.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:34:16.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:34:16.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:34:16.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:34:16.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:34:16.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:34:16.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:34:16.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:34:16.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:16.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:34:16.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:34:16.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:34:16.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:34:16.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:16.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:34:16.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:34:16.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:34:16.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:34:16.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:34:16.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:34:16.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:34:16.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:34:16.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.085 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.086 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:16.086 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:16.086 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.179614887, 2024-04-24T14:34:16.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:34:16.086 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.086 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:16.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:16.086 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:34:16.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:16.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:34:16.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:34:16.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:34:16.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:34:16.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:34:16.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:34:16.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:34:16.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:34:16.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:16.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:34:16.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:34:16.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:34:16.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:34:16.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:16.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:34:16.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:34:16.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.179595135, 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:16.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:16.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:16.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:34:16.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:34:16.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:34:16.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:34:16.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:34:16.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:34:16.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:34:16.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:34:16.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:16.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:34:16.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:34:16.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:34:16.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:34:16.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:34:16.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:34:16.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.097 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.097 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:16.097 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:16.098 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.179577688, 2024-04-24T14:34:16.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:34:16.098 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.098 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:16.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:16.098 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:34:16.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.098 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:16.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:16.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:16.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:34:16.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:34:16.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:34:16.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:34:16.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:34:16.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:34:16.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:34:16.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:34:16.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:16.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:34:16.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:34:16.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:34:16.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:34:16.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:16.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:34:16.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:34:16.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:34:16.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:34:16.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:34:16.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:34:16.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:34:16.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:34:16.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.103 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4d9853ae:::rbd_id.INVALIDIMAGE:head, version: 53'12, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4d9853ae:::rbd_id.INVALIDIMAGE:head@53'12, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:16.104 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:16.104 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.154299157000001, 2024-04-24T14:34:16.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075925020000000001, 2024-04-24T14:34:16.104 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.104 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.104 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:16.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:16.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905243+0000", 2024-04-24T14:34:16.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905253+0000", 2024-04-24T14:34:16.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1619999999999997e-06 2024-04-24T14:34:16.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905262+0000", 2024-04-24T14:34:16.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2399999999999996e-06 2024-04-24T14:34:16.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905268+0000", 2024-04-24T14:34:16.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2530000000000001e-06 2024-04-24T14:34:16.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912457+0000", 2024-04-24T14:34:16.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071890779999999998 2024-04-24T14:34:16.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912462+0000", 2024-04-24T14:34:16.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5159999999999996e-06 2024-04-24T14:34:16.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912836+0000", 2024-04-24T14:34:16.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037484699999999998 2024-04-24T14:34:16.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.107 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.108 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 62'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@62'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:16.108 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:16.108 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.130680227999999, 2024-04-24T14:34:16.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036829689999999999, 2024-04-24T14:34:16.108 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.108 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.108 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:16.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928862+0000", 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930264+0000", 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014016059999999999 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930281+0000", 2024-04-24T14:34:16.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7859999999999998e-05 2024-04-24T14:34:16.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930291+0000", 2024-04-24T14:34:16.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9299999999999998e-06 2024-04-24T14:34:16.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930894+0000", 2024-04-24T14:34:16.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060250899999999999 2024-04-24T14:34:16.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930903+0000", 2024-04-24T14:34:16.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4599999999999992e-06 2024-04-24T14:34:16.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.932546+0000", 2024-04-24T14:34:16.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016425019999999999 2024-04-24T14:34:16.111 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.111 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.111 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.112 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head, version: 62'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head@62'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:16.112 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:16.112 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.12496528, 2024-04-24T14:34:16.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037783690000000002, 2024-04-24T14:34:16.112 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.112 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.112 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:16.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:16.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934576+0000", 2024-04-24T14:34:16.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:16.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936033+0000", 2024-04-24T14:34:16.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001456658 2024-04-24T14:34:16.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936071+0000", 2024-04-24T14:34:16.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7752e-05 2024-04-24T14:34:16.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936087+0000", 2024-04-24T14:34:16.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5919000000000001e-05 2024-04-24T14:34:16.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936226+0000", 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013910800000000001 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936240+0000", 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.456e-05 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938356+0000", 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021157540000000001 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.116 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 62'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@62'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:16.116 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:16.116 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.123918752, 2024-04-24T14:34:16.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052678259999999998, 2024-04-24T14:34:16.116 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.116 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.116 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:16.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:16.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935623+0000", 2024-04-24T14:34:16.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937055+0000", 2024-04-24T14:34:16.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014314480000000001 2024-04-24T14:34:16.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937085+0000", 2024-04-24T14:34:16.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0204000000000001e-05 2024-04-24T14:34:16.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937097+0000", 2024-04-24T14:34:16.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1708e-05 2024-04-24T14:34:16.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938410+0000", 2024-04-24T14:34:16.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013128910000000001 2024-04-24T14:34:16.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938419+0000", 2024-04-24T14:34:16.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1740000000000006e-06 2024-04-24T14:34:16.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.940892+0000", 2024-04-24T14:34:16.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024734990000000001 2024-04-24T14:34:16.119 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.119 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.119 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.120 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 62'82, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@62'82, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:16.120 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:16.120 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.114407361, 2024-04-24T14:34:16.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003877447, 2024-04-24T14:34:16.120 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.120 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.120 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:16.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:16.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945134+0000", 2024-04-24T14:34:16.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:16.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946323+0000", 2024-04-24T14:34:16.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001188635 2024-04-24T14:34:16.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946352+0000", 2024-04-24T14:34:16.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9238e-05 2024-04-24T14:34:16.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946374+0000", 2024-04-24T14:34:16.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1396999999999999e-05 2024-04-24T14:34:16.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.947098+0000", 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072458199999999998 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.947109+0000", 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0557000000000001e-05 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.949013+0000", 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019044929999999999 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.123 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.124 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.124 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:16.124 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:16.124 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.107950376, 2024-04-24T14:34:16.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041579369999999996, 2024-04-24T14:34:16.124 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.124 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.124 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:16.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951585+0000", 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999928 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953339+0000", 2024-04-24T14:34:16.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017540019999999999 2024-04-24T14:34:16.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953363+0000", 2024-04-24T14:34:16.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3178000000000001e-05 2024-04-24T14:34:16.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953374+0000", 2024-04-24T14:34:16.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1015e-05 2024-04-24T14:34:16.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953522+0000", 2024-04-24T14:34:16.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014857800000000001 2024-04-24T14:34:16.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953565+0000", 2024-04-24T14:34:16.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2511e-05 2024-04-24T14:34:16.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955751+0000", 2024-04-24T14:34:16.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002185958 2024-04-24T14:34:16.127 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.127 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.127 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.128 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 62'82, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@62'82, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:16.128 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:16.128 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.10753586, 2024-04-24T14:34:16.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005349749, 2024-04-24T14:34:16.128 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.128 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.128 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:16.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:16.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952006+0000", 2024-04-24T14:34:16.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953689+0000", 2024-04-24T14:34:16.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00168326 2024-04-24T14:34:16.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953717+0000", 2024-04-24T14:34:16.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7994e-05 2024-04-24T14:34:16.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953746+0000", 2024-04-24T14:34:16.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8059999999999999e-05 2024-04-24T14:34:16.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955798+0000", 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002052089 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955812+0000", 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.957357+0000", 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001545537 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.132 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:16.132 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:16.132 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.098745742, 2024-04-24T14:34:16.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051567999999999996, 2024-04-24T14:34:16.132 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.132 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.132 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:16.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:16.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960796+0000", 2024-04-24T14:34:16.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962459+0000", 2024-04-24T14:34:16.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016622259999999999 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962481+0000", 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2359999999999999e-05 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962496+0000", 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.523e-05 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.964299+0000", 2024-04-24T14:34:16.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018025129999999999 2024-04-24T14:34:16.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.964316+0000", 2024-04-24T14:34:16.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7436999999999999e-05 2024-04-24T14:34:16.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.965954+0000", 2024-04-24T14:34:16.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016381099999999999 2024-04-24T14:34:16.135 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.135 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.135 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.136 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head, version: 62'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head@62'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:16.136 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:16.136 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.065739553, 2024-04-24T14:34:16.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003994821, 2024-04-24T14:34:16.136 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.136 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.136 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:16.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993802+0000", 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994846+0000", 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00104368 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994884+0000", 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7796000000000002e-05 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994900+0000", 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5917e-05 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995046+0000", 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014653499999999999 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995055+0000", 2024-04-24T14:34:16.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0759999999999992e-06 2024-04-24T14:34:16.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.997798+0000", 2024-04-24T14:34:16.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027432709999999998 2024-04-24T14:34:16.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.139 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.139 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(2.2 68/67 [PushOp(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 49'1, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@49'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.065432095, 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038339310000000001, 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:16.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994110+0000", 2024-04-24T14:34:16.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994914+0000", 2024-04-24T14:34:16.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080383099999999997 2024-04-24T14:34:16.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994943+0000", 2024-04-24T14:34:16.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9475000000000002e-05 2024-04-24T14:34:16.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994955+0000", 2024-04-24T14:34:16.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1525000000000001e-05 2024-04-24T14:34:16.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995123+0000", 2024-04-24T14:34:16.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016812899999999999 2024-04-24T14:34:16.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995143+0000", 2024-04-24T14:34:16.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9724999999999998e-05 2024-04-24T14:34:16.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.997945+0000", 2024-04-24T14:34:16.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028023890000000002 2024-04-24T14:34:16.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.143 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.143 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head, version: 62'144, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head@62'144, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:16.143 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.0251269, 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041651270000000002, 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034415+0000", 2024-04-24T14:34:16.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:16.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035025+0000", 2024-04-24T14:34:16.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060999700000000003 2024-04-24T14:34:16.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035077+0000", 2024-04-24T14:34:16.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2132999999999997e-05 2024-04-24T14:34:16.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035100+0000", 2024-04-24T14:34:16.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2690000000000001e-05 2024-04-24T14:34:16.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035293+0000", 2024-04-24T14:34:16.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019358900000000001 2024-04-24T14:34:16.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035311+0000", 2024-04-24T14:34:16.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7952999999999999e-05 2024-04-24T14:34:16.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.038581+0000", 2024-04-24T14:34:16.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032700670000000002 2024-04-24T14:34:16.147 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.147 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.147 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.147 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head, version: 62'144, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head@62'144, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:16.147 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.010961464999999, 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039835970000000002, 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048581+0000", 2024-04-24T14:34:16.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049792+0000", 2024-04-24T14:34:16.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012111579999999999 2024-04-24T14:34:16.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049823+0000", 2024-04-24T14:34:16.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1137999999999999e-05 2024-04-24T14:34:16.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049838+0000", 2024-04-24T14:34:16.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5061e-05 2024-04-24T14:34:16.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.050005+0000", 2024-04-24T14:34:16.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016745400000000001 2024-04-24T14:34:16.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.050022+0000", 2024-04-24T14:34:16.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7022000000000001e-05 2024-04-24T14:34:16.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.052565+0000", 2024-04-24T14:34:16.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025428669999999999 2024-04-24T14:34:16.151 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.151 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.151 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.151 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.151 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.151 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:16.153 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:16.155+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:16.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:16.253 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:16.255+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:16.286 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:34:16.289 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:34:16.294 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:16.294 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:16.295 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:16.295 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:16.300 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-04-24T14:34:15.980010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":216597525,"num_objects":91,"num_object_clones":2,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":8,"num_read":4395,"num_read_kb":167075,"num_write":750,"num_write_kb":267831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":205521400,"num_keys_recovered":68,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":751,"ondisk_log_size":751,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":1927192,"kb_used_data":423380,"kb_used_omap":11,"kb_used_meta":1503732,"kb_avail":469932008,"statfs":{"total":483183820800,"available":481210376192,"internally_reserved":0,"allocated":433541120,"data_stored":430835230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11359,"internal_metadata":1539822497},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":76,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":198,"num_read_kb":2654,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043788"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":55,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:10.149782+0000","last_change":"2024-04-24T14:34:06.850979+0000","last_active":"2024-04-24T14:34:10.149782+0000","last_peered":"2024-04-24T14:34:10.149782+0000","last_clean":"2024-04-24T14:34:10.149782+0000","last_became_active":"2024-04-24T14:34:06.850634+0000","last_became_peered":"2024-04-24T14:34:06.850634+0000","last_unstale":"2024-04-24T14:34:10.149782+0000","last_undegraded":"2024-04-24T14:34:10.149782+0000","last_fullsized":"2024-04-24T14:34:10.149782+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:00.068704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062826099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":220,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:10.149964+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:34:10.149964+0000","last_peered":"2024-04-24T14:34:10.149964+0000","last_clean":"2024-04-24T14:34:10.149964+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:34:10.149964+0000","last_undegraded":"2024-04-24T14:34:10.149964+0000","last_fullsized":"2024-04-24T14:34:10.149964+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"49'3","reported_seq":77,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:10.150076+0000","last_change":"2024-04-24T14:34:02.957063+0000","last_active":"2024-04-24T14:34:10.150076+0000","last_peered":"2024-04-24T14:34:10.150076+0000","last_clean":"2024-04-24T14:34:10.150076+0000","last_became_active":"2024-04-24T14:34:02.882014+0000","last_became_peered":"2024-04-24T14:34:02.882014+0000","last_unstale":"2024-04-24T14:34:10.150076+0000","last_undegraded":"2024-04-24T14:34:10.150076+0000","last_fullsized":"2024-04-24T14:34:10.150076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:36:04.594048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014822850000000001,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"76'84","reported_seq":371,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.219296+0000","last_change":"2024-04-24T14:34:02.974227+0000","last_active":"2024-04-24T14:34:13.219296+0000","last_peered":"2024-04-24T14:34:13.219296+0000","last_clean":"2024-04-24T14:34:13.219296+0000","last_became_active":"2024-04-24T14:34:02.878704+0000","last_became_peered":"2024-04-24T14:34:02.878704+0000","last_unstale":"2024-04-24T14:34:13.219296+0000","last_undegraded":"2024-04-24T14:34:13.219296+0000","last_fullsized":"2024-04-24T14:34:13.219296+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:14:17.858654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022812169999999999,"stat_sum":{"num_bytes":20971593,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":958,"num_read_kb":43545,"num_write":72,"num_write_kb":32772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":20971555,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.6","version":"74'1","reported_seq":46,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:09.830172+0000","last_change":"2024-04-24T14:34:01.880601+0000","last_active":"2024-04-24T14:34:09.830172+0000","last_peered":"2024-04-24T14:34:09.830172+0000","last_clean":"2024-04-24T14:34:09.830172+0000","last_became_active":"2024-04-24T14:34:01.879862+0000","last_became_peered":"2024-04-24T14:34:01.879862+0000","last_unstale":"2024-04-24T14:34:09.830172+0000","last_undegraded":"2024-04-24T14:34:09.830172+0000","last_fullsized":"2024-04-24T14:34:09.830172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:31:46.206212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039681800000000001,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"49'3","reported_seq":98,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:10.149650+0000","last_change":"2024-04-24T14:34:02.001524+0000","last_active":"2024-04-24T14:34:10.149650+0000","last_peered":"2024-04-24T14:34:10.149650+0000","last_clean":"2024-04-24T14:34:10.149650+0000","last_became_active":"2024-04-24T14:34:01.880423+0000","last_became_peered":"2024-04-24T14:34:01.880423+0000","last_unstale":"2024-04-24T14:34:10.149650+0000","last_undegraded":"2024-04-24T14:34:10.149650+0000","last_fullsized":"2024-04-24T14:34:10.149650+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:56:32.827897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079506599999999998,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"76'249","reported_seq":1784,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-24T14:34:11.190475+0000","last_change":"2024-04-24T14:34:09.834110+0000","last_active":"2024-04-24T14:34:11.190475+0000","last_peered":"2024-04-24T14:34:11.190475+0000","last_clean":"2024-04-24T14:34:11.190475+0000","last_became_active":"2024-04-24T14:34:09.833171+0000","last_became_peered":"2024-04-24T14:34:09.833171+0000","last_unstale":"2024-04-24T14:34:11.190475+0000","last_undegraded":"2024-04-24T14:34:11.190475+0000","last_fullsized":"2024-04-24T14:34:11.190475+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:34:29.046015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079543300000000001,"stat_sum":{"num_bytes":27263014,"num_objects":11,"num_object_clones":2,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":2089,"num_read_kb":67755,"num_write":170,"num_write_kb":52231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":25165862,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.219068+0000","last_change":"2024-04-24T14:34:01.878005+0000","last_active":"2024-04-24T14:34:13.219068+0000","last_peered":"2024-04-24T14:34:13.219068+0000","last_clean":"2024-04-24T14:34:13.219068+0000","last_became_active":"2024-04-24T14:34:01.877166+0000","last_became_peered":"2024-04-24T14:34:01.877166+0000","last_unstale":"2024-04-24T14:34:13.219068+0000","last_undegraded":"2024-04-24T14:34:13.219068+0000","last_fullsized":"2024-04-24T14:34:13.219068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:16:28.973324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047250999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"63'4","reported_seq":107,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:10.149951+0000","last_change":"2024-04-24T14:34:01.925168+0000","last_active":"2024-04-24T14:34:10.149951+0000","last_peered":"2024-04-24T14:34:10.149951+0000","last_clean":"2024-04-24T14:34:10.149951+0000","last_became_active":"2024-04-24T14:34:01.881101+0000","last_became_peered":"2024-04-24T14:34:01.881101+0000","last_unstale":"2024-04-24T14:34:10.149951+0000","last_undegraded":"2024-04-24T14:34:10.149951+0000","last_fullsized":"2024-04-24T14:34:10.149951+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:19:53.594540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000475358,"stat_sum":{"num_bytes":8388646,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388646,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"76'167","reported_seq":512,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.218931+0000","last_change":"2024-04-24T14:34:04.833510+0000","last_active":"2024-04-24T14:34:13.218931+0000","last_peered":"2024-04-24T14:34:13.218931+0000","last_clean":"2024-04-24T14:34:13.218931+0000","last_became_active":"2024-04-24T14:34:04.832060+0000","last_became_peered":"2024-04-24T14:34:04.832060+0000","last_unstale":"2024-04-24T14:34:13.218931+0000","last_undegraded":"2024-04-24T14:34:13.218931+0000","last_fullsized":"2024-04-24T14:34:13.218931+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:05:32.595970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044041099999999998,"stat_sum":{"num_bytes":20971558,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":689,"num_read_kb":38400,"num_write":223,"num_write_kb":32904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":20971558,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.4","version":"74'1","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.219028+0000","last_change":"2024-04-24T14:34:01.880675+0000","last_active":"2024-04-24T14:34:13.219028+0000","last_peered":"2024-04-24T14:34:13.219028+0000","last_clean":"2024-04-24T14:34:13.219028+0000","last_became_active":"2024-04-24T14:34:01.880072+0000","last_became_peered":"2024-04-24T14:34:01.880072+0000","last_unstale":"2024-04-24T14:34:13.219028+0000","last_undegraded":"2024-04-24T14:34:13.219028+0000","last_fullsized":"2024-04-24T14:34:13.219028+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:50.701297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032986300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"49'5","reported_seq":37,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.219047+0000","last_change":"2024-04-24T14:34:03.105395+0000","last_active":"2024-04-24T14:34:13.219047+0000","last_peered":"2024-04-24T14:34:13.219047+0000","last_clean":"2024-04-24T14:34:13.219047+0000","last_became_active":"2024-04-24T14:34:02.879139+0000","last_became_peered":"2024-04-24T14:34:02.879139+0000","last_unstale":"2024-04-24T14:34:13.219047+0000","last_undegraded":"2024-04-24T14:34:13.219047+0000","last_fullsized":"2024-04-24T14:34:13.219047+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:06:37.465800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086195800000000002,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"70'84","reported_seq":33,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:09.828179+0000","last_change":"2024-04-24T14:34:06.971354+0000","last_active":"2024-04-24T14:34:09.828179+0000","last_peered":"2024-04-24T14:34:09.828179+0000","last_clean":"2024-04-24T14:34:09.828179+0000","last_became_active":"2024-04-24T14:34:06.852598+0000","last_became_peered":"2024-04-24T14:34:06.852598+0000","last_unstale":"2024-04-24T14:34:09.828179+0000","last_undegraded":"2024-04-24T14:34:09.828179+0000","last_fullsized":"2024-04-24T14:34:09.828179+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:34:56.872354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052296799999999996,"stat_sum":{"num_bytes":12583004,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":463,"num_read_kb":16975,"num_write":78,"num_write_kb":16420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777362,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.2","version":"53'2","reported_seq":41,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.219218+0000","last_change":"2024-04-24T14:34:06.856414+0000","last_active":"2024-04-24T14:34:13.219218+0000","last_peered":"2024-04-24T14:34:13.219218+0000","last_clean":"2024-04-24T14:34:13.219218+0000","last_became_active":"2024-04-24T14:34:06.851987+0000","last_became_peered":"2024-04-24T14:34:06.851987+0000","last_unstale":"2024-04-24T14:34:13.219218+0000","last_undegraded":"2024-04-24T14:34:13.219218+0000","last_fullsized":"2024-04-24T14:34:13.219218+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:48:02.402734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054416299999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"49'12","reported_seq":64,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.219151+0000","last_change":"2024-04-24T14:34:02.094718+0000","last_active":"2024-04-24T14:34:13.219151+0000","last_peered":"2024-04-24T14:34:13.219151+0000","last_clean":"2024-04-24T14:34:13.219151+0000","last_became_active":"2024-04-24T14:34:01.876957+0000","last_became_peered":"2024-04-24T14:34:01.876957+0000","last_unstale":"2024-04-24T14:34:13.219151+0000","last_undegraded":"2024-04-24T14:34:13.219151+0000","last_fullsized":"2024-04-24T14:34:13.219151+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:00:15.749808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070345800000000001,"stat_sum":{"num_bytes":37748774,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":14,"num_read_kb":135,"num_write":12,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":37748774,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":43,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:09.830658+0000","last_change":"2024-04-24T14:34:01.880865+0000","last_active":"2024-04-24T14:34:09.830658+0000","last_peered":"2024-04-24T14:34:09.830658+0000","last_clean":"2024-04-24T14:34:09.830658+0000","last_became_active":"2024-04-24T14:34:01.880253+0000","last_became_peered":"2024-04-24T14:34:01.880253+0000","last_unstale":"2024-04-24T14:34:09.830658+0000","last_undegraded":"2024-04-24T14:34:09.830658+0000","last_fullsized":"2024-04-24T14:34:09.830658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:34:54.353042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030291500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"49'5","reported_seq":88,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:10.149879+0000","last_change":"2024-04-24T14:33:34.702742+0000","last_active":"2024-04-24T14:34:10.149879+0000","last_peered":"2024-04-24T14:34:10.149879+0000","last_clean":"2024-04-24T14:34:10.149879+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:34:10.149879+0000","last_undegraded":"2024-04-24T14:34:10.149879+0000","last_fullsized":"2024-04-24T14:34:10.149879+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064337400000000005,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":128,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'1","reported_seq":56,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:09.830129+0000","last_change":"2024-04-24T14:34:06.850978+0000","last_active":"2024-04-24T14:34:09.830129+0000","last_peered":"2024-04-24T14:34:09.830129+0000","last_clean":"2024-04-24T14:34:09.830129+0000","last_became_active":"2024-04-24T14:34:06.850755+0000","last_became_peered":"2024-04-24T14:34:06.850755+0000","last_unstale":"2024-04-24T14:34:09.830129+0000","last_undegraded":"2024-04-24T14:34:09.830129+0000","last_fullsized":"2024-04-24T14:34:09.830129+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:44:46.603392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013794250000000001,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"76'5","reported_seq":51,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.219018+0000","last_change":"2024-04-24T14:34:02.162119+0000","last_active":"2024-04-24T14:34:13.219018+0000","last_peered":"2024-04-24T14:34:13.219018+0000","last_clean":"2024-04-24T14:34:13.219018+0000","last_became_active":"2024-04-24T14:34:01.877290+0000","last_became_peered":"2024-04-24T14:34:01.877290+0000","last_unstale":"2024-04-24T14:34:13.219018+0000","last_undegraded":"2024-04-24T14:34:13.219018+0000","last_fullsized":"2024-04-24T14:34:13.219018+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:29.247345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059660799999999999,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582912,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"74'3","reported_seq":29,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:09.828214+0000","last_change":"2024-04-24T14:34:06.852245+0000","last_active":"2024-04-24T14:34:09.828214+0000","last_peered":"2024-04-24T14:34:09.828214+0000","last_clean":"2024-04-24T14:34:09.828214+0000","last_became_active":"2024-04-24T14:34:06.850685+0000","last_became_peered":"2024-04-24T14:34:06.850685+0000","last_unstale":"2024-04-24T14:34:09.828214+0000","last_undegraded":"2024-04-24T14:34:09.828214+0000","last_fullsized":"2024-04-24T14:34:09.828214+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:11.360282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000856414,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"64'5","reported_seq":123,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:10.149852+0000","last_change":"2024-04-24T14:34:06.924942+0000","last_active":"2024-04-24T14:34:10.149852+0000","last_peered":"2024-04-24T14:34:10.149852+0000","last_clean":"2024-04-24T14:34:10.149852+0000","last_became_active":"2024-04-24T14:34:06.852864+0000","last_became_peered":"2024-04-24T14:34:06.852864+0000","last_unstale":"2024-04-24T14:34:10.149852+0000","last_undegraded":"2024-04-24T14:34:10.149852+0000","last_fullsized":"2024-04-24T14:34:10.149852+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:42.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023873739999999998,"stat_sum":{"num_bytes":8388684,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":5,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":16,"acting":16,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":81789169,"num_objects":42,"num_object_clones":2,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":8,"num_read":4199,"num_read_kb":166675,"num_write":543,"num_write_kb":134327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":83886337,"num_keys_recovered":40,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184623104,"data_stored":197132805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":584,"ondisk_log_size":584,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":134217918,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":42,"num_read_kb":270,"num_write":42,"num_write_kb":131082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":121635006,"num_keys_recovered":28,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":302030848,"data_stored":301990268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640283,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640283,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347392,"kb_used_data":45940,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94024448,"statfs":{"total":96636764160,"available":96281034752,"internally_reserved":0,"allocated":47042560,"data_stored":46500070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.873},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.63200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":2,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.72199999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":0.59199999999999997}]},{"osd":5,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.78000000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":376628,"kb_used_data":75176,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93995212,"statfs":{"total":96636764160,"available":96251097088,"internally_reserved":0,"allocated":76980224,"data_stored":76450417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.86899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":2,"last update":"Wed Apr 24 14:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":1.0629999999999999}]},{"osd":4,"last update":"Wed Apr 24 14:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.91300000000000003}]},{"osd":5,"last update":"Wed Apr 24 14:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.71399999999999997}]}]},{"osd":2,"up_from":11,"seq":47244640283,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":343272,"kb_used_data":41820,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94028568,"statfs":{"total":96636764160,"available":96285253632,"internally_reserved":0,"allocated":42823680,"data_stored":42305582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.748}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":3,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":1.111}]},{"osd":4,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.995}]},{"osd":5,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.82799999999999996}]}]},{"osd":1,"up_from":66,"seq":283467841540,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":442300,"kb_used_data":144296,"kb_used_omap":4,"kb_used_meta":297979,"kb_avail":93929540,"statfs":{"total":96636764160,"available":96183848960,"internally_reserved":0,"allocated":147759104,"data_stored":147185596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305131073},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640282,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417600,"kb_used_data":116148,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93954240,"statfs":{"total":96636764160,"available":96209141760,"internally_reserved":0,"allocated":118935552,"data_stored":118393565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Wed Apr 24 14:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.745},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.626}]},{"osd":3,"last update":"Wed Apr 24 14:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.94899999999999995}]},{"osd":4,"last update":"Wed Apr 24 14:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.0089999999999999}]},{"osd":5,"last update":"Wed Apr 24 14:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.98699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79695872,"data_stored":79691814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104869888,"data_stored":104857714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20975616,"data_stored":20971558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54538240,"data_stored":54526066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39862272,"data_stored":46137474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41959424,"data_stored":41943151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20975616,"data_stored":20971558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20983808,"data_stored":20971593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":39862272,"data_stored":46137474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20979712,"data_stored":20971555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":51,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:34:16.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:34:16.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:16.353 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:16.359+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:16.443 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:16.443 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:16.443 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:16.443 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:16.444 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:16.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:16.454 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:16.459+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:16.547 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:16.547 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:16.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:16.547 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:16.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.547 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:97 3.1 3:990e5750:::rbd_header.12ff64619e2d:head [watch reconnect cookie 140276853203408 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:16.547 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:16.547 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.197340244999999, 2024-04-24T14:34:16.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012161632, 2024-04-24T14:34:16.547 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.547 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.548 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 97 2024-04-24T14:34:16.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.548 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:16.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:16.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352830+0000", 2024-04-24T14:34:16.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352834+0000", 2024-04-24T14:34:16.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2699999999999998e-06 2024-04-24T14:34:16.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352844+0000", 2024-04-24T14:34:16.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0406e-05 2024-04-24T14:34:16.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352850+0000", 2024-04-24T14:34:16.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6439999999999997e-06 2024-04-24T14:34:16.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362391+0000", 2024-04-24T14:34:16.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095408969999999996 2024-04-24T14:34:16.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362683+0000", 2024-04-24T14:34:16.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029238000000000002 2024-04-24T14:34:16.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.364992+0000", 2024-04-24T14:34:16.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023092020000000002 2024-04-24T14:34:16.551 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.551 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.551 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.552 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:225 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:16.552 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:16.552 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.044898132999997, 2024-04-24T14:34:16.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010959015000000001, 2024-04-24T14:34:16.552 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.552 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.552 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 225 2024-04-24T14:34:16.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.552 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:16.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:16.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:16.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505275+0000", 2024-04-24T14:34:16.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0269999999999998e-06 2024-04-24T14:34:16.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505279+0000", 2024-04-24T14:34:16.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4279999999999998e-06 2024-04-24T14:34:16.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505282+0000", 2024-04-24T14:34:16.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.982e-06 2024-04-24T14:34:16.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515209+0000", 2024-04-24T14:34:16.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099266419999999994 2024-04-24T14:34:16.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515466+0000", 2024-04-24T14:34:16.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025719700000000002 2024-04-24T14:34:16.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.516232+0000", 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076608899999999996 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.793866391000002, 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:16.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:34:16.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:34:16.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:34:16.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:34:16.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:34:16.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:34:16.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:34:16.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:16.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:34:16.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:34:16.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.562 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:16.562 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:16.562 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.793815557000002, 2024-04-24T14:34:16.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:34:16.562 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.562 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.562 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:34:16.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.562 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:16.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:34:16.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:34:16.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:34:16.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:34:16.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:34:16.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:34:16.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.79377925, 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.567 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:34:16.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:34:16.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:34:16.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:34:16.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:34:16.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:34:16.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:34:16.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:34:16.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:34:16.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:16.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:34:16.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:34:16.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.790368000000001, 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:16.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:16.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:34:16.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:16.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:34:16.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:34:16.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:16.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:34:16.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:34:16.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:34:16.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:34:16.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:16.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:34:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:34:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:34:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:34:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:34:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:34:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:34:16.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:34:16.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.578 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.578 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:16.578 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:16.578 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.790306323000003, 2024-04-24T14:34:16.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:34:16.578 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.578 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.578 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.579 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:34:16.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.579 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:16.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:16.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:34:16.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:34:16.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:34:16.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:34:16.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:34:16.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:34:16.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:34:16.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:34:16.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:34:16.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:16.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:34:16.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:34:16.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:34:16.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:34:16.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:16.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:34:16.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:34:16.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:34:16.583 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.790271746999998, 2024-04-24T14:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.584 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:16.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:34:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:34:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:34:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:34:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:34:16.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:34:16.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:16.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:34:16.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:34:16.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:34:16.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:34:16.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.790242407999997, 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:34:16.589 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.590 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.590 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.590 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.590 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.590 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:34:16.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.590 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:16.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:34:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:34:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:34:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:34:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:34:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:34:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.594 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.790165825999999, 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:34:16.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:34:16.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:16.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:34:16.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:34:16.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:34:16.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:34:16.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:34:16.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:34:16.600 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.600 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.600 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.600 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:16.600 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:16.600 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.790118280999998, 2024-04-24T14:34:16.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:34:16.600 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.600 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:16.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:34:16.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:34:16.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:34:16.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:34:16.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:34:16.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:34:16.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.606 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:16.606 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:16.606 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.790077639000003, 2024-04-24T14:34:16.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:34:16.606 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.606 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.606 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.606 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.606 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.606 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:34:16.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.606 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:16.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:16.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:16.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:34:16.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:34:16.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:34:16.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:34:16.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:34:16.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:34:16.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:34:16.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:34:16.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:16.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:34:16.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:34:16.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:34:16.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:34:16.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:16.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:34:16.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:34:16.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:34:16.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:34:16.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:34:16.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:34:16.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:34:16.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:34:16.611 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.611 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.611 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.790060527999998, 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:16.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:16.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:34:16.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:34:16.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:34:16.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:34:16.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:34:16.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:34:16.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:34:16.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:34:16.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:34:16.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:16.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:34:16.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:34:16.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:34:16.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:34:16.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:34:16.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:34:16.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:34:16.617 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.617 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.617 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.617 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:481 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:16.617 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:16.617 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.756188821000002, 2024-04-24T14:34:16.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010661718000000001, 2024-04-24T14:34:16.617 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.618 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.618 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 481 2024-04-24T14:34:16.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.618 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:16.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793981+0000", 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793985+0000", 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8469999999999998e-06 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0220000000000004e-06 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793998+0000", 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8559999999999998e-06 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803807+0000", 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098088629999999993 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803916+0000", 2024-04-24T14:34:16.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010854899999999999 2024-04-24T14:34:16.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.804644+0000", 2024-04-24T14:34:16.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072835399999999996 2024-04-24T14:34:16.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.621 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.621 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:497 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:16.621 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:16.622 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.738248509000002, 2024-04-24T14:34:16.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013996471, 2024-04-24T14:34:16.622 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.622 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:16.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:16.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:16.622 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 497 2024-04-24T14:34:16.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.622 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811922+0000", 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811925+0000", 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4120000000000001e-06 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811936+0000", 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1077000000000001e-05 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811941+0000", 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7539999999999997e-06 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822289+0000", 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010347635000000001 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822457+0000", 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000168257 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.825919+0000", 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003461979 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:25 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.582690568, 2024-04-24T14:34:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013349789000000001, 2024-04-24T14:34:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 25 2024-04-24T14:34:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:16.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:16.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967479+0000", 2024-04-24T14:34:16.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967669+0000", 2024-04-24T14:34:16.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000189829 2024-04-24T14:34:16.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967687+0000", 2024-04-24T14:34:16.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8199999999999999e-05 2024-04-24T14:34:16.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967695+0000", 2024-04-24T14:34:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4869999999999996e-06 2024-04-24T14:34:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.972501+0000", 2024-04-24T14:34:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048057509999999996 2024-04-24T14:34:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.972824+0000", 2024-04-24T14:34:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032322299999999998 2024-04-24T14:34:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.973071+0000", 2024-04-24T14:34:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024751299999999999 2024-04-24T14:34:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977624+0000", 2024-04-24T14:34:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045527570000000002 2024-04-24T14:34:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980627+0000", 2024-04-24T14:34:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003002972 2024-04-24T14:34:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980743+0000", 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980830+0000", 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7534999999999995e-05 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:26 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:16.631 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.582466833000002, 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013403597999999999, 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 26 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967703+0000", 2024-04-24T14:34:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967874+0000", 2024-04-24T14:34:16.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017088499999999999 2024-04-24T14:34:16.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967886+0000", 2024-04-24T14:34:16.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1744e-05 2024-04-24T14:34:16.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967893+0000", 2024-04-24T14:34:16.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2024-04-24T14:34:16.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974028+0000", 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061356650000000002 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974347+0000", 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031897400000000003 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974774+0000", 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042661700000000001 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:16.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980930+0000", 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061563360000000001 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980945+0000", 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4864e-05 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981035+0000", 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9390999999999998e-05 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981108+0000", 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3255000000000003e-05 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:27 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.582269687, 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013394262000000001, 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:16.637 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 27 2024-04-24T14:34:16.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.638 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:16.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:16.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967900+0000", 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968061+0000", 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016101900000000001 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968072+0000", 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1012999999999999e-05 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968079+0000", 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.46e-06 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.976820+0000", 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087406289999999998 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.976951+0000", 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131187 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977129+0000", 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000178135 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981126+0000", 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00399709 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981193+0000", 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7083999999999997e-05 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:16.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981251+0000", 2024-04-24T14:34:16.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7754000000000003e-05 2024-04-24T14:34:16.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981296+0000", 2024-04-24T14:34:16.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4765000000000003e-05 2024-04-24T14:34:16.642 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.642 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.642 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.642 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:28 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:16.642 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.582084455, 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013427247999999999, 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 28 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968086+0000", 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968247+0000", 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016121199999999999 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968260+0000", 2024-04-24T14:34:16.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.275e-05 2024-04-24T14:34:16.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968266+0000", 2024-04-24T14:34:16.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3770000000000001e-06 2024-04-24T14:34:16.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977699+0000", 2024-04-24T14:34:16.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094331439999999992 2024-04-24T14:34:16.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977894+0000", 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194945 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.978258+0000", 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036359400000000002 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981140+0000", 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028821540000000001 2024-04-24T14:34:16.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981360+0000", 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022056300000000001 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981444+0000", 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3733999999999995e-05 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981514+0000", 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9702000000000005e-05 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.647 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:29 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.581897609999999, 2024-04-24T14:34:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014492283, 2024-04-24T14:34:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 29 2024-04-24T14:34:16.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.649 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:16.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:16.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:16.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:16.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:16.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:16.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:16.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:16.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:16.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:16.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968434+0000", 2024-04-24T14:34:16.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016130199999999999 2024-04-24T14:34:16.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:16.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968484+0000", 2024-04-24T14:34:16.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0003000000000002e-05 2024-04-24T14:34:16.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968499+0000", 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4628e-05 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979299+0000", 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010800624999999999 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979488+0000", 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000189037 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979784+0000", 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029562000000000002 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982305+0000", 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025212339999999998 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982680+0000", 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000374762 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:16.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982731+0000", 2024-04-24T14:34:16.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1146999999999997e-05 2024-04-24T14:34:16.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:16.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:16.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:16.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982766+0000", 2024-04-24T14:34:16.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4755000000000002e-05 2024-04-24T14:34:16.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.653 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:16.653 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:16.653 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:16.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:16.656 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:34:16.657 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:34:16.657 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-04-24T14:34:15.980010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":216597525,"num_objects":91,"num_object_clones":2,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":8,"num_read":4395,"num_read_kb":167075,"num_write":750,"num_write_kb":267831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":205521400,"num_keys_recovered":68,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":751,"ondisk_log_size":751,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":1927192,"kb_used_data":423380,"kb_used_omap":11,"kb_used_meta":1503732,"kb_avail":469932008,"statfs":{"total":483183820800,"available":481210376192,"internally_reserved":0,"allocated":433541120,"data_stored":430835230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11359,"internal_metadata":1539822497},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":76,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":198,"num_read_kb":2654,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043788"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":55,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:10.149782+0000","last_change":"2024-04-24T14:34:06.850979+0000","last_active":"2024-04-24T14:34:10.149782+0000","last_peered":"2024-04-24T14:34:10.149782+0000","last_clean":"2024-04-24T14:34:10.149782+0000","last_became_active":"2024-04-24T14:34:06.850634+0000","last_became_peered":"2024-04-24T14:34:06.850634+0000","last_unstale":"2024-04-24T14:34:10.149782+0000","last_undegraded":"2024-04-24T14:34:10.149782+0000","last_fullsized":"2024-04-24T14:34:10.149782+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:11:00.068704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062826099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":220,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:10.149964+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:34:10.149964+0000","last_peered":"2024-04-24T14:34:10.149964+0000","last_clean":"2024-04-24T14:34:10.149964+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:34:10.149964+0000","last_undegraded":"2024-04-24T14:34:10.149964+0000","last_fullsized":"2024-04-24T14:34:10.149964+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"49'3","reported_seq":77,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:10.150076+0000","last_change":"2024-04-24T14:34:02.957063+0000","last_active":"2024-04-24T14:34:10.150076+0000","last_peered":"2024-04-24T14:34:10.150076+0000","last_clean":"2024-04-24T14:34:10.150076+0000","last_became_active":"2024-04-24T14:34:02.882014+0000","last_became_peered":"2024-04-24T14:34:02.882014+0000","last_unstale":"2024-04-24T14:34:10.150076+0000","last_undegraded":"2024-04-24T14:34:10.150076+0000","last_fullsized":"2024-04-24T14:34:10.150076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:36:04.594048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014822850000000001,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"76'84","reported_seq":371,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.219296+0000","last_change":"2024-04-24T14:34:02.974227+0000","last_active":"2024-04-24T14:34:13.219296+0000","last_peered":"2024-04-24T14:34:13.219296+0000","last_clean":"2024-04-24T14:34:13.219296+0000","last_became_active":"2024-04-24T14:34:02.878704+0000","last_became_peered":"2024-04-24T14:34:02.878704+0000","last_unstale":"2024-04-24T14:34:13.219296+0000","last_undegraded":"2024-04-24T14:34:13.219296+0000","last_fullsized":"2024-04-24T14:34:13.219296+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:14:17.858654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022812169999999999,"stat_sum":{"num_bytes":20971593,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":958,"num_read_kb":43545,"num_write":72,"num_write_kb":32772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":20971555,"num_keys_recovered":3,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.6","version":"74'1","reported_seq":46,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:09.830172+0000","last_change":"2024-04-24T14:34:01.880601+0000","last_active":"2024-04-24T14:34:09.830172+0000","last_peered":"2024-04-24T14:34:09.830172+0000","last_clean":"2024-04-24T14:34:09.830172+0000","last_became_active":"2024-04-24T14:34:01.879862+0000","last_became_peered":"2024-04-24T14:34:01.879862+0000","last_unstale":"2024-04-24T14:34:09.830172+0000","last_undegraded":"2024-04-24T14:34:09.830172+0000","last_fullsized":"2024-04-24T14:34:09.830172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:31:46.206212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039681800000000001,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"49'3","reported_seq":98,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:10.149650+0000","last_change":"2024-04-24T14:34:02.001524+0000","last_active":"2024-04-24T14:34:10.149650+0000","last_peered":"2024-04-24T14:34:10.149650+0000","last_clean":"2024-04-24T14:34:10.149650+0000","last_became_active":"2024-04-24T14:34:01.880423+0000","last_became_peered":"2024-04-24T14:34:01.880423+0000","last_unstale":"2024-04-24T14:34:10.149650+0000","last_undegraded":"2024-04-24T14:34:10.149650+0000","last_fullsized":"2024-04-24T14:34:10.149650+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:56:32.827897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079506599999999998,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"76'249","reported_seq":1784,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-24T14:34:11.190475+0000","last_change":"2024-04-24T14:34:09.834110+0000","last_active":"2024-04-24T14:34:11.190475+0000","last_peered":"2024-04-24T14:34:11.190475+0000","last_clean":"2024-04-24T14:34:11.190475+0000","last_became_active":"2024-04-24T14:34:09.833171+0000","last_became_peered":"2024-04-24T14:34:09.833171+0000","last_unstale":"2024-04-24T14:34:11.190475+0000","last_undegraded":"2024-04-24T14:34:11.190475+0000","last_fullsized":"2024-04-24T14:34:11.190475+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:34:29.046015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079543300000000001,"stat_sum":{"num_bytes":27263014,"num_objects":11,"num_object_clones":2,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":2089,"num_read_kb":67755,"num_write":170,"num_write_kb":52231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":25165862,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.219068+0000","last_change":"2024-04-24T14:34:01.878005+0000","last_active":"2024-04-24T14:34:13.219068+0000","last_peered":"2024-04-24T14:34:13.219068+0000","last_clean":"2024-04-24T14:34:13.219068+0000","last_became_active":"2024-04-24T14:34:01.877166+0000","last_became_peered":"2024-04-24T14:34:01.877166+0000","last_unstale":"2024-04-24T14:34:13.219068+0000","last_undegraded":"2024-04-24T14:34:13.219068+0000","last_fullsized":"2024-04-24T14:34:13.219068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:16:28.973324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047250999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"63'4","reported_seq":107,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:10.149951+0000","last_change":"2024-04-24T14:34:01.925168+0000","last_active":"2024-04-24T14:34:10.149951+0000","last_peered":"2024-04-24T14:34:10.149951+0000","last_clean":"2024-04-24T14:34:10.149951+0000","last_became_active":"2024-04-24T14:34:01.881101+0000","last_became_peered":"2024-04-24T14:34:01.881101+0000","last_unstale":"2024-04-24T14:34:10.149951+0000","last_undegraded":"2024-04-24T14:34:10.149951+0000","last_fullsized":"2024-04-24T14:34:10.149951+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:19:53.594540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000475358,"stat_sum":{"num_bytes":8388646,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":4,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388646,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"76'167","reported_seq":512,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.218931+0000","last_change":"2024-04-24T14:34:04.833510+0000","last_active":"2024-04-24T14:34:13.218931+0000","last_peered":"2024-04-24T14:34:13.218931+0000","last_clean":"2024-04-24T14:34:13.218931+0000","last_became_active":"2024-04-24T14:34:04.832060+0000","last_became_peered":"2024-04-24T14:34:04.832060+0000","last_unstale":"2024-04-24T14:34:13.218931+0000","last_undegraded":"2024-04-24T14:34:13.218931+0000","last_fullsized":"2024-04-24T14:34:13.218931+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:05:32.595970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044041099999999998,"stat_sum":{"num_bytes":20971558,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":689,"num_read_kb":38400,"num_write":223,"num_write_kb":32904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":20971558,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.4","version":"74'1","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.219028+0000","last_change":"2024-04-24T14:34:01.880675+0000","last_active":"2024-04-24T14:34:13.219028+0000","last_peered":"2024-04-24T14:34:13.219028+0000","last_clean":"2024-04-24T14:34:13.219028+0000","last_became_active":"2024-04-24T14:34:01.880072+0000","last_became_peered":"2024-04-24T14:34:01.880072+0000","last_unstale":"2024-04-24T14:34:13.219028+0000","last_undegraded":"2024-04-24T14:34:13.219028+0000","last_fullsized":"2024-04-24T14:34:13.219028+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:50.701297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032986300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"49'5","reported_seq":37,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.219047+0000","last_change":"2024-04-24T14:34:03.105395+0000","last_active":"2024-04-24T14:34:13.219047+0000","last_peered":"2024-04-24T14:34:13.219047+0000","last_clean":"2024-04-24T14:34:13.219047+0000","last_became_active":"2024-04-24T14:34:02.879139+0000","last_became_peered":"2024-04-24T14:34:02.879139+0000","last_unstale":"2024-04-24T14:34:13.219047+0000","last_undegraded":"2024-04-24T14:34:13.219047+0000","last_fullsized":"2024-04-24T14:34:13.219047+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:06:37.465800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086195800000000002,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"70'84","reported_seq":33,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:09.828179+0000","last_change":"2024-04-24T14:34:06.971354+0000","last_active":"2024-04-24T14:34:09.828179+0000","last_peered":"2024-04-24T14:34:09.828179+0000","last_clean":"2024-04-24T14:34:09.828179+0000","last_became_active":"2024-04-24T14:34:06.852598+0000","last_became_peered":"2024-04-24T14:34:06.852598+0000","last_unstale":"2024-04-24T14:34:09.828179+0000","last_undegraded":"2024-04-24T14:34:09.828179+0000","last_fullsized":"2024-04-24T14:34:09.828179+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:34:56.872354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052296799999999996,"stat_sum":{"num_bytes":12583004,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":463,"num_read_kb":16975,"num_write":78,"num_write_kb":16420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777362,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.2","version":"53'2","reported_seq":41,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.219218+0000","last_change":"2024-04-24T14:34:06.856414+0000","last_active":"2024-04-24T14:34:13.219218+0000","last_peered":"2024-04-24T14:34:13.219218+0000","last_clean":"2024-04-24T14:34:13.219218+0000","last_became_active":"2024-04-24T14:34:06.851987+0000","last_became_peered":"2024-04-24T14:34:06.851987+0000","last_unstale":"2024-04-24T14:34:13.219218+0000","last_undegraded":"2024-04-24T14:34:13.219218+0000","last_fullsized":"2024-04-24T14:34:13.219218+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:48:02.402734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054416299999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"49'12","reported_seq":64,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.219151+0000","last_change":"2024-04-24T14:34:02.094718+0000","last_active":"2024-04-24T14:34:13.219151+0000","last_peered":"2024-04-24T14:34:13.219151+0000","last_clean":"2024-04-24T14:34:13.219151+0000","last_became_active":"2024-04-24T14:34:01.876957+0000","last_became_peered":"2024-04-24T14:34:01.876957+0000","last_unstale":"2024-04-24T14:34:13.219151+0000","last_undegraded":"2024-04-24T14:34:13.219151+0000","last_fullsized":"2024-04-24T14:34:13.219151+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:00:15.749808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070345800000000001,"stat_sum":{"num_bytes":37748774,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":14,"num_read_kb":135,"num_write":12,"num_write_kb":36867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":37748774,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":43,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:09.830658+0000","last_change":"2024-04-24T14:34:01.880865+0000","last_active":"2024-04-24T14:34:09.830658+0000","last_peered":"2024-04-24T14:34:09.830658+0000","last_clean":"2024-04-24T14:34:09.830658+0000","last_became_active":"2024-04-24T14:34:01.880253+0000","last_became_peered":"2024-04-24T14:34:01.880253+0000","last_unstale":"2024-04-24T14:34:09.830658+0000","last_undegraded":"2024-04-24T14:34:09.830658+0000","last_fullsized":"2024-04-24T14:34:09.830658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:34:54.353042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030291500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"49'5","reported_seq":88,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:10.149879+0000","last_change":"2024-04-24T14:33:34.702742+0000","last_active":"2024-04-24T14:34:10.149879+0000","last_peered":"2024-04-24T14:34:10.149879+0000","last_clean":"2024-04-24T14:34:10.149879+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:34:10.149879+0000","last_undegraded":"2024-04-24T14:34:10.149879+0000","last_fullsized":"2024-04-24T14:34:10.149879+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064337400000000005,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":128,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'1","reported_seq":56,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:09.830129+0000","last_change":"2024-04-24T14:34:06.850978+0000","last_active":"2024-04-24T14:34:09.830129+0000","last_peered":"2024-04-24T14:34:09.830129+0000","last_clean":"2024-04-24T14:34:09.830129+0000","last_became_active":"2024-04-24T14:34:06.850755+0000","last_became_peered":"2024-04-24T14:34:06.850755+0000","last_unstale":"2024-04-24T14:34:09.830129+0000","last_undegraded":"2024-04-24T14:34:09.830129+0000","last_fullsized":"2024-04-24T14:34:09.830129+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:44:46.603392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013794250000000001,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"76'5","reported_seq":51,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-24T14:34:13.219018+0000","last_change":"2024-04-24T14:34:02.162119+0000","last_active":"2024-04-24T14:34:13.219018+0000","last_peered":"2024-04-24T14:34:13.219018+0000","last_clean":"2024-04-24T14:34:13.219018+0000","last_became_active":"2024-04-24T14:34:01.877290+0000","last_became_peered":"2024-04-24T14:34:01.877290+0000","last_unstale":"2024-04-24T14:34:13.219018+0000","last_undegraded":"2024-04-24T14:34:13.219018+0000","last_fullsized":"2024-04-24T14:34:13.219018+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:29.247345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059660799999999999,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":12582912,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"74'3","reported_seq":29,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:09.828214+0000","last_change":"2024-04-24T14:34:06.852245+0000","last_active":"2024-04-24T14:34:09.828214+0000","last_peered":"2024-04-24T14:34:09.828214+0000","last_clean":"2024-04-24T14:34:09.828214+0000","last_became_active":"2024-04-24T14:34:06.850685+0000","last_became_peered":"2024-04-24T14:34:06.850685+0000","last_unstale":"2024-04-24T14:34:09.828214+0000","last_undegraded":"2024-04-24T14:34:09.828214+0000","last_fullsized":"2024-04-24T14:34:09.828214+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:11.360282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000856414,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"64'5","reported_seq":123,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-24T14:34:10.149852+0000","last_change":"2024-04-24T14:34:06.924942+0000","last_active":"2024-04-24T14:34:10.149852+0000","last_peered":"2024-04-24T14:34:10.149852+0000","last_clean":"2024-04-24T14:34:10.149852+0000","last_became_active":"2024-04-24T14:34:06.852864+0000","last_became_peered":"2024-04-24T14:34:06.852864+0000","last_unstale":"2024-04-24T14:34:10.149852+0000","last_undegraded":"2024-04-24T14:34:10.149852+0000","last_fullsized":"2024-04-24T14:34:10.149852+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:42.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023873739999999998,"stat_sum":{"num_bytes":8388684,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":5,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":16,"acting":16,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":81789169,"num_objects":42,"num_object_clones":2,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":8,"num_read":4199,"num_read_kb":166675,"num_write":543,"num_write_kb":134327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":83886337,"num_keys_recovered":40,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":40,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184623104,"data_stored":197132805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":584,"ondisk_log_size":584,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":134217918,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":42,"num_read_kb":270,"num_write":42,"num_write_kb":131082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":121635006,"num_keys_recovered":28,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":302030848,"data_stored":301990268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640283,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640283,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347392,"kb_used_data":45940,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94024448,"statfs":{"total":96636764160,"available":96281034752,"internally_reserved":0,"allocated":47042560,"data_stored":46500070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.873},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.63200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":2,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.72199999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":0.59199999999999997}]},{"osd":5,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.78000000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":376628,"kb_used_data":75176,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93995212,"statfs":{"total":96636764160,"available":96251097088,"internally_reserved":0,"allocated":76980224,"data_stored":76450417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.86899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":2,"last update":"Wed Apr 24 14:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":1.0629999999999999}]},{"osd":4,"last update":"Wed Apr 24 14:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.91300000000000003}]},{"osd":5,"last update":"Wed Apr 24 14:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.71399999999999997}]}]},{"osd":2,"up_from":11,"seq":47244640283,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":343272,"kb_used_data":41820,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94028568,"statfs":{"total":96636764160,"available":96285253632,"internally_reserved":0,"allocated":42823680,"data_stored":42305582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.748}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":3,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":1.111}]},{"osd":4,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.995}]},{"osd":5,"last update":"Wed Apr 24 14:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.82799999999999996}]}]},{"osd":1,"up_from":66,"seq":283467841540,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":442300,"kb_used_data":144296,"kb_used_omap":4,"kb_used_meta":297979,"kb_avail":93929540,"statfs":{"total":96636764160,"available":96183848960,"internally_reserved":0,"allocated":147759104,"data_stored":147185596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305131073},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640282,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417600,"kb_used_data":116148,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93954240,"statfs":{"total":96636764160,"available":96209141760,"internally_reserved":0,"allocated":118935552,"data_stored":118393565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Wed Apr 24 14:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.745},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.626}]},{"osd":3,"last update":"Wed Apr 24 14:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.94899999999999995}]},{"osd":4,"last update":"Wed Apr 24 14:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.0089999999999999}]},{"osd":5,"last update":"Wed Apr 24 14:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.98699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79695872,"data_stored":79691814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104869888,"data_stored":104857714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20975616,"data_stored":20971558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54538240,"data_stored":54526066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39862272,"data_stored":46137474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41959424,"data_stored":41943151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20975616,"data_stored":20971558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20983808,"data_stored":20971593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":39862272,"data_stored":46137474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20979712,"data_stored":20971555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":51,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:34:16.659 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T14:34:16.659 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:16.659+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:16.660 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:34:16.660 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-04-24T14:34:16.660 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-04-24T14:34:16.755 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:16.759+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:16.759 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:16.759 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:16.760 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:16.760 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:16.768 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:16.856 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:16.859+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:16.910 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:16.910 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:16.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:16.958 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:16.961+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:16.961 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-04-24T14:34:16.963+0000 7ff9c25de700 -1 mon.a@0(leader).osd e81 definitely_dead 0 2024-04-24T14:34:16.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T14:34:16.986 INFO:teuthology.orchestra.run.smithi090.stderr:marked in osd.5. 2024-04-24T14:34:16.997 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-04-24T14:34:17.032 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:17.033 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:17.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:17.033 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:17.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.033 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:192 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:17.033 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:17.033 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.566132805000002, 2024-04-24T14:34:17.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010036329, 2024-04-24T14:34:17.033 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.033 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.034 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.034 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.034 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192 2024-04-24T14:34:17.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.034 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:17.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:17.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471099+0000", 2024-04-24T14:34:17.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471103+0000", 2024-04-24T14:34:17.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8800000000000001e-06 2024-04-24T14:34:17.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471111+0000", 2024-04-24T14:34:17.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2930000000000001e-06 2024-04-24T14:34:17.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471116+0000", 2024-04-24T14:34:17.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1800000000000004e-06 2024-04-24T14:34:17.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480338+0000", 2024-04-24T14:34:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092213069999999998 2024-04-24T14:34:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480467+0000", 2024-04-24T14:34:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012901500000000001 2024-04-24T14:34:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481136+0000", 2024-04-24T14:34:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066948400000000003 2024-04-24T14:34:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.038 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.038 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.038 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:17.038 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:17.038 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.566109769000001, 2024-04-24T14:34:17.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:34:17.038 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.038 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.039 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.039 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.039 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.039 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:34:17.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.039 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:17.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:17.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:34:17.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:34:17.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:34:17.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:34:17.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:34:17.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:34:17.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:34:17.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:34:17.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:34:17.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:34:17.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:34:17.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:34:17.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:34:17.043 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.043 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.043 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.043 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.043 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:17.043 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.190753065999999, 2024-04-24T14:34:17.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:34:17.043 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.043 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.043 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.043 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.044 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.044 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:34:17.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.044 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:17.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:17.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:34:17.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:34:17.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:34:17.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:34:17.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:34:17.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:34:17.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:17.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:34:17.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:34:17.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:34:17.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:34:17.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:34:17.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:34:17.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.048 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.048 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.048 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:526 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.048 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:17.048 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.189365543000001, 2024-04-24T14:34:17.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010568783999999999, 2024-04-24T14:34:17.048 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.048 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.048 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.048 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.048 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.049 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 526 2024-04-24T14:34:17.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.049 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:17.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:17.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847866+0000", 2024-04-24T14:34:17.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847871+0000", 2024-04-24T14:34:17.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1590000000000001e-06 2024-04-24T14:34:17.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847883+0000", 2024-04-24T14:34:17.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.172e-05 2024-04-24T14:34:17.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847890+0000", 2024-04-24T14:34:17.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6950000000000003e-06 2024-04-24T14:34:17.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857418+0000", 2024-04-24T14:34:17.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095285909999999995 2024-04-24T14:34:17.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857588+0000", 2024-04-24T14:34:17.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169541 2024-04-24T14:34:17.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858436+0000", 2024-04-24T14:34:17.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084814700000000005 2024-04-24T14:34:17.052 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.052 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.053 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.053 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.053 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:17.053 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.189324912000004, 2024-04-24T14:34:17.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:34:17.053 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.053 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.054 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.054 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:34:17.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.054 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:17.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:17.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:34:17.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:34:17.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:34:17.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:34:17.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:34:17.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:34:17.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:34:17.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:34:17.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:34:17.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:34:17.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:34:17.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:34:17.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:34:17.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.058 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.058 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.058 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:17.058 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.189297822999997, 2024-04-24T14:34:17.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:34:17.058 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.058 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.058 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.059 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.059 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.059 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:34:17.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.059 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:17.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:17.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:34:17.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:17.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:34:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:34:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:34:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:34:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:34:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:34:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:34:17.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:34:17.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:34:17.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:34:17.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:34:17.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:34:17.063 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.063 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.063 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.063 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.063 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:17.063 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.189273856, 2024-04-24T14:34:17.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:34:17.063 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.063 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.064 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.064 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:34:17.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.064 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:17.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:17.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:34:17.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:17.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:34:17.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:34:17.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:34:17.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:34:17.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:34:17.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:34:17.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:34:17.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:34:17.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:34:17.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:34:17.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:34:17.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:34:17.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.068 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.068 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.068 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.068 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:17.068 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.175289876999997, 2024-04-24T14:34:17.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:34:17.068 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.068 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.069 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.069 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:34:17.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.069 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:17.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:17.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:34:17.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:17.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:34:17.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:34:17.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:34:17.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:34:17.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:34:17.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:34:17.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:34:17.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:34:17.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:34:17.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:34:17.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:34:17.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:34:17.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:34:17.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:34:17.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:34:17.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:34:17.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:34:17.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:34:17.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:34:17.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:34:17.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.074 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.075 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.075 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:17.075 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.175173565999998, 2024-04-24T14:34:17.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:34:17.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.075 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:34:17.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.075 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:17.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:17.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:34:17.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:17.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:34:17.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:34:17.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:34:17.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:34:17.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:34:17.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:34:17.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:34:17.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:34:17.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:34:17.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:34:17.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:34:17.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:34:17.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:34:17.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:34:17.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:34:17.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:34:17.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:34:17.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:34:17.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:34:17.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:34:17.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.081 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.081 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.081 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.081 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:17.081 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.175125907999998, 2024-04-24T14:34:17.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:34:17.081 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.081 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.081 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.081 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.082 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.082 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:34:17.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.082 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:17.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:17.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:34:17.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:34:17.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:34:17.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:34:17.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:34:17.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:34:17.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:34:17.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:34:17.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:34:17.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:34:17.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:34:17.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:34:17.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:34:17.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:34:17.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:34:17.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:34:17.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:34:17.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:34:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:34:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:34:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:34:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.088 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.088 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:17.088 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.171088609999998, 2024-04-24T14:34:17.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:34:17.088 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.088 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.088 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.088 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.088 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.088 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:34:17.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.088 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:17.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:17.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:34:17.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:17.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:34:17.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:34:17.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:34:17.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:34:17.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:34:17.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:34:17.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:17.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:34:17.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:34:17.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:34:17.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:34:17.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:34:17.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:34:17.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:34:17.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:34:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:34:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:34:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:34:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:34:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:34:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.094 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.094 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.094 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:17.094 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.171024850999999, 2024-04-24T14:34:17.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:34:17.094 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.094 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.094 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.095 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.095 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.095 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:34:17.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.095 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:17.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:17.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:34:17.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:34:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:34:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:34:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:34:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:34:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:34:17.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:34:17.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:34:17.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:34:17.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:34:17.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:34:17.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:34:17.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:34:17.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:34:17.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:34:17.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:34:17.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:34:17.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:34:17.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:34:17.100 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.101 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.101 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.101 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.101 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:17.101 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.170999361, 2024-04-24T14:34:17.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:34:17.101 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.101 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.102 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.102 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:34:17.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.102 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:17.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:17.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:17.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:17.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:17.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:34:17.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:34:17.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:34:17.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:34:17.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:34:17.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:34:17.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:34:17.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:34:17.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:34:17.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:34:17.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:34:17.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:34:17.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:34:17.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:34:17.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:34:17.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:34:17.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:34:17.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:34:17.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:34:17.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.107 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.107 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.108 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:17.108 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.170975611000003, 2024-04-24T14:34:17.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:34:17.108 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.108 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.108 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:34:17.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.108 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:17.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:17.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:34:17.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:34:17.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:34:17.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:34:17.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:34:17.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:34:17.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:34:17.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:34:17.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:34:17.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:34:17.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:34:17.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:34:17.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:34:17.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:34:17.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:34:17.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:34:17.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:34:17.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:34:17.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:34:17.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:34:17.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:34:17.115 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.115 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.115 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.115 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.115 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:17.115 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.170937098000003, 2024-04-24T14:34:17.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:34:17.116 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.116 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.116 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.116 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.116 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.116 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:34:17.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.116 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:17.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:17.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:34:17.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:17.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:34:17.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:34:17.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:34:17.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:34:17.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:34:17.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:34:17.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:34:17.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:34:17.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:34:17.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:34:17.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:34:17.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:34:17.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:34:17.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:34:17.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:34:17.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:34:17.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:34:17.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:34:17.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:34:17.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:34:17.121 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.121 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.121 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.122 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.122 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:17.122 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.170908900999997, 2024-04-24T14:34:17.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:34:17.122 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.122 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.122 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.122 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.122 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.122 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:34:17.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.123 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:17.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:17.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:34:17.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:17.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:34:17.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:34:17.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:34:17.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:34:17.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:34:17.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:34:17.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:34:17.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:34:17.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:34:17.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:34:17.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:34:17.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:34:17.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:34:17.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:34:17.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:34:17.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:34:17.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:34:17.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:34:17.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:34:17.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:34:17.128 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.128 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.128 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.128 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.128 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:17.128 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.170887444000002, 2024-04-24T14:34:17.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:34:17.128 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.128 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.128 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.128 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.129 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.129 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:34:17.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.129 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:17.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:17.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:34:17.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:34:17.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:34:17.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:34:17.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:34:17.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:34:17.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:17.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:34:17.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:34:17.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:34:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:34:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:34:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:34:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:34:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:34:17.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:34:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:34:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:34:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:34:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:34:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:34:17.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.134 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.134 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.134 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:17.134 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.170868601999999, 2024-04-24T14:34:17.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:34:17.134 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.135 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.135 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:34:17.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.135 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:17.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:17.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:34:17.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:17.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:34:17.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:34:17.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:34:17.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:34:17.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:34:17.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:34:17.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:34:17.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:34:17.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:34:17.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:34:17.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:34:17.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:34:17.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:34:17.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:34:17.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:34:17.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:34:17.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:561 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.128905981999999, 2024-04-24T14:34:17.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01066806, 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 561 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:17.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908326+0000", 2024-04-24T14:34:17.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:17.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908329+0000", 2024-04-24T14:34:17.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.802e-06 2024-04-24T14:34:17.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908338+0000", 2024-04-24T14:34:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6260000000000001e-06 2024-04-24T14:34:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908345+0000", 2024-04-24T14:34:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1389999999999998e-06 2024-04-24T14:34:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.917978+0000", 2024-04-24T14:34:17.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096329840000000007 2024-04-24T14:34:17.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918178+0000", 2024-04-24T14:34:17.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019999199999999999 2024-04-24T14:34:17.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918995+0000", 2024-04-24T14:34:17.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081708099999999999 2024-04-24T14:34:17.145 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.145 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.145 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.145 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:17.145 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:17.145 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.958036452999998, 2024-04-24T14:34:17.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:34:17.145 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:17.145 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:17.146 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:17.146 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:34:17.146 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:17.146 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:34:17.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.146 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:17.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:17.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:17.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:17.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:34:17.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:17.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:17.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:34:17.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:34:17.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:17.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:34:17.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:17.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:34:17.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:34:17.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:34:17.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:34:17.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:34:17.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:34:17.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:34:17.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:34:17.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:34:17.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:34:17.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:17.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:34:17.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:34:17.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:17.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:34:17.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:34:17.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:17.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:17.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:17.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:34:17.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:34:17.155 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.155 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.155 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.155 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:17.156 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:17.156 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:17.158 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:17.159 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:17.161+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:17.259 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:17.265+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:17.271 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:17.272 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:17.272 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:17.272 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:17.281 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:17.360 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:17.365+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:17.384 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:17.384 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:17.384 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:17.393 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:17.460 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:17.463+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:17.539 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:17.539 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:17.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:17.540 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:17.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.540 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:17.540 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:17.540 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 114.918366358, 2024-04-24T14:34:17.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:34:17.540 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.540 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:17.540 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.540 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:17.540 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:17.541 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:34:17.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.541 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:17.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:17.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:34:17.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:17.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:34:17.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:34:17.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:34:17.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:34:17.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:34:17.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:34:17.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:34:17.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:34:17.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:34:17.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:34:17.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:17.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:34:17.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:34:17.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:17.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:34:17.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:34:17.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:17.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:34:17.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:34:17.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:17.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:34:17.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:34:17.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:34:17.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:34:17.546 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.546 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.546 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.546 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:17.546 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:17.546 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 114.918189271, 2024-04-24T14:34:17.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:34:17.546 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.546 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:17.547 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.547 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:17.547 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:17.547 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:34:17.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.547 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:17.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:17.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:34:17.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:17.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:34:17.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:34:17.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:34:17.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:34:17.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:34:17.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:34:17.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:34:17.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:34:17.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:34:17.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:34:17.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:17.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:34:17.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:34:17.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:17.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:34:17.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:34:17.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:17.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:34:17.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:34:17.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:17.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:34:17.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:34:17.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:34:17.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:34:17.552 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.552 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.552 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.552 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:17.552 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:17.552 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 114.918085357, 2024-04-24T14:34:17.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:34:17.552 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.552 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:17.553 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.553 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:17.553 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:17.553 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:34:17.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.553 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:17.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:17.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:34:17.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:17.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:34:17.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:34:17.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:34:17.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:34:17.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:34:17.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:17.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:34:17.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:34:17.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:34:17.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:34:17.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:17.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:34:17.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:34:17.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:17.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:34:17.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:34:17.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:17.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:34:17.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:34:17.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:17.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:34:17.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:34:17.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:34:17.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:34:17.558 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.558 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.558 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.558 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:17.559 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:17.559 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 114.917696906, 2024-04-24T14:34:17.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:34:17.559 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.559 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:17.559 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.559 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:17.559 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:17.559 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:34:17.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.559 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:34:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:34:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:34:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:34:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:34:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:34:17.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:34:17.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:34:17.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:34:17.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:34:17.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:34:17.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:17.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:34:17.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:34:17.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:17.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:34:17.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:34:17.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:17.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:34:17.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:34:17.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:17.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:34:17.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:34:17.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:34:17.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:34:17.564 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.564 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.564 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 114.917611694, 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:17.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:17.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:34:17.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:17.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:34:17.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:34:17.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:34:17.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:34:17.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:34:17.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:34:17.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:34:17.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:34:17.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:34:17.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:34:17.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:17.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:34:17.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:34:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:34:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:34:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:34:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:34:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:17.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:34:17.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:34:17.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:34:17.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:34:17.570 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.570 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.570 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.570 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:17.570 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 114.917374813, 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:17.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:34:17.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:17.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:34:17.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:34:17.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:34:17.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:34:17.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:34:17.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:34:17.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:34:17.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:34:17.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:34:17.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:34:17.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:17.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:34:17.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:34:17.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:17.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:34:17.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:34:17.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:17.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:34:17.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:34:17.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:17.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:34:17.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:34:17.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:34:17.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:34:17.576 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.576 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.576 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.576 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:17.576 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:17.576 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 114.91729329099999, 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:17.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:34:17.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:17.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:34:17.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:34:17.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:34:17.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:34:17.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:34:17.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:34:17.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:34:17.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:34:17.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:34:17.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:34:17.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:17.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:34:17.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:34:17.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:17.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:34:17.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:34:17.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:17.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:34:17.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:34:17.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:17.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:34:17.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:34:17.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:34:17.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:34:17.582 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.582 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.582 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.582 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:17.582 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:17.582 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 114.485380944, 2024-04-24T14:34:17.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:34:17.583 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.583 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:17.583 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.583 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:17.583 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:17.583 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:34:17.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.583 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:17.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:17.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:34:17.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:34:17.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:34:17.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:34:17.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:34:17.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:34:17.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:34:17.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:34:17.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:34:17.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:34:17.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:34:17.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:17.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:34:17.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:34:17.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:17.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:34:17.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:34:17.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:17.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:34:17.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:34:17.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:17.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:34:17.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:34:17.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:34:17.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:34:17.588 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.588 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.588 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.588 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:17.588 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:17.588 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 114.485352644, 2024-04-24T14:34:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:34:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:34:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:17.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:34:17.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:17.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:34:17.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:34:17.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:34:17.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:34:17.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:34:17.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:34:17.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:34:17.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:34:17.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:34:17.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:34:17.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:17.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:34:17.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:34:17.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:17.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:34:17.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:34:17.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:17.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:34:17.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:34:17.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:17.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:34:17.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:34:17.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:34:17.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:34:17.594 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.594 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.594 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.595 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:17.595 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:17.595 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 114.48526493999999, 2024-04-24T14:34:17.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:34:17.595 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.595 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:17.595 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.595 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:17.595 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:17.595 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:34:17.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.595 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:17.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:17.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:34:17.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:34:17.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:34:17.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:34:17.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:34:17.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:34:17.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:34:17.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:34:17.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:34:17.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:34:17.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:34:17.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:17.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:34:17.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:34:17.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:17.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:34:17.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:34:17.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:17.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:34:17.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:34:17.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:17.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:34:17.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:34:17.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:34:17.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:34:17.600 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.600 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.600 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.601 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:17.601 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:17.601 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 114.485142139, 2024-04-24T14:34:17.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:34:17.601 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.601 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:17.601 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.601 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:17.601 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:17.601 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:34:17.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.601 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:17.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:17.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:34:17.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:34:17.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:34:17.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:34:17.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:34:17.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:17.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:34:17.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:34:17.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:17.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:34:17.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:34:17.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:17.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:34:17.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:34:17.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:17.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:34:17.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:34:17.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:34:17.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:34:17.606 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.606 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.607 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.607 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:34:17.607 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:17.607 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.178897579000001, 2024-04-24T14:34:17.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:34:17.607 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.607 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:17.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:34:17.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:34:17.607 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:34:17.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.608 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:17.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:17.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:34:17.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:17.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:34:17.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:34:17.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:34:17.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:34:17.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:34:17.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:34:17.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:34:17.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:34:17.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:34:17.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:34:17.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:34:17.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:34:17.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:34:17.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:17.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:34:17.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:34:17.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:34:17.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:34:17.612 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.612 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.612 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.612 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:17.612 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:17.612 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.484773580000002, 2024-04-24T14:34:17.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:34:17.612 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.612 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:17.613 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.613 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:34:17.613 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:34:17.613 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:34:17.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.613 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:17.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:17.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:34:17.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:17.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:34:17.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:34:17.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:34:17.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:34:17.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:34:17.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:34:17.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:34:17.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:34:17.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:34:17.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:34:17.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:34:17.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:34:17.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:34:17.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:34:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:34:17.618 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.618 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.618 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.618 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:17.618 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:17.618 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.525413614000001, 2024-04-24T14:34:17.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:34:17.618 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.618 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:17.618 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.618 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:17.618 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:17.619 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:34:17.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.619 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:17.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:17.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:34:17.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:34:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:34:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:34:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:34:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:34:17.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:34:17.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:34:17.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:34:17.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:34:17.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:34:17.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:34:17.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:34:17.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:34:17.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:34:17.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:34:17.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:34:17.623 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.623 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.623 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.624 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:17.624 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:17.624 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.525333059000001, 2024-04-24T14:34:17.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:34:17.624 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.624 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:17.624 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.624 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:17.624 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:17.624 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:34:17.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.625 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:17.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:17.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:34:17.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:34:17.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:34:17.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:34:17.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:34:17.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:34:17.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:34:17.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:34:17.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:34:17.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:34:17.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:34:17.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:34:17.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:34:17.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:34:17.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:34:17.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:34:17.629 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.629 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.629 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.630 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:17.630 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:17.630 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.525285329999999, 2024-04-24T14:34:17.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:34:17.630 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.630 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:17.630 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.630 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:17.630 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:17.630 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:34:17.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.630 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:34:17.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:34:17.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:34:17.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:34:17.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:34:17.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:34:17.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:34:17.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:34:17.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:34:17.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:34:17.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:34:17.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:34:17.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:34:17.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:34:17.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:34:17.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:34:17.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:34:17.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:34:17.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:34:17.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:34:17.635 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.635 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.635 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.635 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:17.635 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:17.636 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.525257567000001, 2024-04-24T14:34:17.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:34:17.636 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.636 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:17.636 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.636 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:17.636 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:17.636 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:34:17.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.636 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:17.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:17.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:34:17.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:17.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:34:17.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:34:17.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:34:17.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:34:17.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:34:17.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:34:17.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:34:17.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:34:17.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:34:17.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:34:17.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:34:17.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:34:17.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:34:17.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:34:17.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:34:17.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:34:17.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:34:17.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:34:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:34:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:34:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.525218131999999, 2024-04-24T14:34:17.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:34:17.642 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.642 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:17.642 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.642 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:17.642 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:17.642 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:34:17.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.642 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:17.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:17.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:34:17.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:17.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:34:17.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:34:17.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:34:17.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:34:17.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:34:17.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:34:17.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:34:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:34:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:34:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:34:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:34:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:34:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:34:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:34:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.525193148, 2024-04-24T14:34:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:34:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.647 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:17.648 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.648 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:17.648 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:17.648 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:34:17.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.648 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:17.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:34:17.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:34:17.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:34:17.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:34:17.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:34:17.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:34:17.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:34:17.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:34:17.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:34:17.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:34:17.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:34:17.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:34:17.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:34:17.652 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.653 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.653 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.653 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:17.653 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:17.653 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 18.525167489000001, 2024-04-24T14:34:17.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:34:17.653 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:17.653 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:17.653 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:17.653 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:17.653 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:17.654 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:34:17.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.654 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:17.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:17.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:17.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:17.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:17.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:17.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:17.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:34:17.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:17.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:17.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:34:17.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:34:17.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:17.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:34:17.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:34:17.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:17.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:34:17.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:17.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:34:17.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:34:17.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:17.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:34:17.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:34:17.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:34:17.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:34:17.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:17.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:34:17.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:34:17.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:17.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:34:17.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:34:17.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:17.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:34:17.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:34:17.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:17.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:17.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:17.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:34:17.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:34:17.659 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.659 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.659 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.659 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:17.659 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:17.659 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:17.661 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:17.661 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:17.663+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:17.762 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:17.767+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:17.768 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:17.864 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:17.864 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:17.864 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:17.866 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:17.867 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:17.869+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:17.967 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:17.973+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:17.990 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-24T14:34:18.022 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:18.022 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:18.022 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:18.022 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:18.023 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:18.033 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:18.034 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:34:18.079 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:18.085+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:18.144 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:18.145 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:18.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:18.145 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:18.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.145 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:379 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.145 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:18.145 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.485287337000003, 2024-04-24T14:34:18.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068078410000000002, 2024-04-24T14:34:18.145 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.145 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.145 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.145 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.146 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.146 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 379 2024-04-24T14:34:18.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.146 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:18.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:18.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664238+0000", 2024-04-24T14:34:18.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:18.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664240+0000", 2024-04-24T14:34:18.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6539999999999999e-06 2024-04-24T14:34:18.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664245+0000", 2024-04-24T14:34:18.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2179999999999998e-06 2024-04-24T14:34:18.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664249+0000", 2024-04-24T14:34:18.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2279999999999998e-06 2024-04-24T14:34:18.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670245+0000", 2024-04-24T14:34:18.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0059962430000000001 2024-04-24T14:34:18.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.670400+0000", 2024-04-24T14:34:18.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015535399999999999 2024-04-24T14:34:18.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671046+0000", 2024-04-24T14:34:18.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064591599999999996 2024-04-24T14:34:18.149 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.149 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.149 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.150 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:380 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.150 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:34:18.150 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.485240709999999, 2024-04-24T14:34:18.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076160830000000001, 2024-04-24T14:34:18.150 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.150 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.150 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.150 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.150 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.150 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 380 2024-04-24T14:34:18.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.150 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:34:18.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664285+0000", 2024-04-24T14:34:18.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664284+0000", 2024-04-24T14:34:18.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:18.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664287+0000", 2024-04-24T14:34:18.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9739999999999998e-06 2024-04-24T14:34:18.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664302+0000", 2024-04-24T14:34:18.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5214999999999999e-05 2024-04-24T14:34:18.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664307+0000", 2024-04-24T14:34:18.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6859999999999997e-06 2024-04-24T14:34:18.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671098+0000", 2024-04-24T14:34:18.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067912479999999997 2024-04-24T14:34:18.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671253+0000", 2024-04-24T14:34:18.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015427400000000001 2024-04-24T14:34:18.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671901+0000", 2024-04-24T14:34:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00064845600000000005 2024-04-24T14:34:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.485211165000003, 2024-04-24T14:34:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:34:18.154 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.155 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.155 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.155 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.155 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.155 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:34:18.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.155 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:18.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:18.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:18.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:18.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:34:18.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:34:18.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:34:18.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:34:18.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:34:18.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:34:18.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:34:18.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:34:18.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:34:18.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:34:18.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:34:18.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:34:18.158 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.158 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.159 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.159 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:382 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.159 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:34:18.159 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.482209681999997, 2024-04-24T14:34:18.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067029910000000002, 2024-04-24T14:34:18.159 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.159 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.159 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.159 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.159 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.159 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 382 2024-04-24T14:34:18.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:34:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667316+0000", 2024-04-24T14:34:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667315+0000", 2024-04-24T14:34:18.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:18.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667321+0000", 2024-04-24T14:34:18.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2099999999999998e-06 2024-04-24T14:34:18.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667336+0000", 2024-04-24T14:34:18.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4612e-05 2024-04-24T14:34:18.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667343+0000", 2024-04-24T14:34:18.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0119999999999999e-06 2024-04-24T14:34:18.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672957+0000", 2024-04-24T14:34:18.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056139700000000002 2024-04-24T14:34:18.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.673127+0000", 2024-04-24T14:34:18.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017044599999999999 2024-04-24T14:34:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674019+0000", 2024-04-24T14:34:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00089165499999999999 2024-04-24T14:34:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.482176639000002, 2024-04-24T14:34:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:34:18.164 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.164 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.164 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.164 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.164 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.164 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:34:18.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.164 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:18.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:18.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:18.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:18.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:34:18.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:34:18.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:34:18.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:34:18.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:34:18.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:34:18.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:34:18.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:34:18.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:34:18.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:34:18.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:34:18.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:34:18.167 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.168 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.168 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.168 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.168 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:18.168 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.482152452999998, 2024-04-24T14:34:18.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:34:18.168 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.168 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.168 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.168 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.168 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.169 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:34:18.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.169 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:18.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:18.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:34:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:34:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:34:18.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:34:18.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:34:18.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:34:18.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:34:18.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:34:18.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:34:18.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:34:18.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:34:18.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:34:18.172 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.482135069000002, 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:34:18.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.174 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:18.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:18.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:34:18.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:18.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:34:18.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:34:18.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:34:18.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:34:18.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:34:18.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:34:18.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:34:18.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:34:18.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:34:18.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:34:18.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:34:18.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:34:18.177 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.177 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.177 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.177 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:399 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.177 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:34:18.177 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.467724934000003, 2024-04-24T14:34:18.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071317810000000002, 2024-04-24T14:34:18.178 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.178 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.178 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.178 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.178 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.178 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 399 2024-04-24T14:34:18.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.178 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:34:18.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681801+0000", 2024-04-24T14:34:18.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681800+0000", 2024-04-24T14:34:18.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:18.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681804+0000", 2024-04-24T14:34:18.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.715e-06 2024-04-24T14:34:18.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681816+0000", 2024-04-24T14:34:18.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1851000000000001e-05 2024-04-24T14:34:18.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681820+0000", 2024-04-24T14:34:18.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6450000000000004e-06 2024-04-24T14:34:18.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688059+0000", 2024-04-24T14:34:18.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062390090000000002 2024-04-24T14:34:18.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688196+0000", 2024-04-24T14:34:18.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013721999999999999 2024-04-24T14:34:18.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688933+0000", 2024-04-24T14:34:18.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073605000000000003 2024-04-24T14:34:18.181 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.182 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.182 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.182 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.182 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:18.182 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.467698976999998, 2024-04-24T14:34:18.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:34:18.182 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.182 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.182 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.182 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.182 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.182 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:34:18.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.183 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.183 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:18.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.183 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:18.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:34:18.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:18.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.184 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:34:18.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:34:18.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.184 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:34:18.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:34:18.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:34:18.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:34:18.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:34:18.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:34:18.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:34:18.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:34:18.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:34:18.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:34:18.186 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.186 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.186 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.186 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.186 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:18.186 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.467666678000001, 2024-04-24T14:34:18.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:34:18.187 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.187 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.187 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.187 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.187 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.187 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:34:18.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.187 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:18.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:18.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:34:18.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:18.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.189 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:34:18.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:34:18.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.189 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:34:18.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:34:18.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.189 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:34:18.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:34:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:34:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:34:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:34:18.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:34:18.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:34:18.191 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.191 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.191 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.191 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:414 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1572864~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.191 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:34:18.191 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.453471268000001, 2024-04-24T14:34:18.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069877910000000001, 2024-04-24T14:34:18.192 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.192 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.192 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.192 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.192 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.192 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 414 2024-04-24T14:34:18.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.192 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:34:18.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696054+0000", 2024-04-24T14:34:18.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696053+0000", 2024-04-24T14:34:18.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:18.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.193 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.193 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696058+0000", 2024-04-24T14:34:18.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7509999999999999e-06 2024-04-24T14:34:18.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696074+0000", 2024-04-24T14:34:18.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5743000000000002e-05 2024-04-24T14:34:18.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696093+0000", 2024-04-24T14:34:18.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9508e-05 2024-04-24T14:34:18.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702314+0000", 2024-04-24T14:34:18.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0062207010000000004 2024-04-24T14:34:18.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.702431+0000", 2024-04-24T14:34:18.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011723299999999999 2024-04-24T14:34:18.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703042+0000", 2024-04-24T14:34:18.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000610752 2024-04-24T14:34:18.195 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.196 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.196 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.196 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.196 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:18.196 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.453414401000003, 2024-04-24T14:34:18.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:34:18.196 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.196 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.196 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.196 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.196 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.197 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:34:18.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.197 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:18.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:18.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:34:18.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:18.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:34:18.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:34:18.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:34:18.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:34:18.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:34:18.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:34:18.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:34:18.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:34:18.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:34:18.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:34:18.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.200 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:34:18.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:34:18.200 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.200 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.200 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.200 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.200 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:18.201 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.453385300999997, 2024-04-24T14:34:18.201 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:34:18.201 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.201 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.201 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.201 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.201 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.201 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:34:18.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.201 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.201 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.201 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:18.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.202 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.202 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:18.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.202 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.202 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:18.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:18.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.202 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:34:18.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:34:18.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:34:18.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:34:18.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:34:18.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:34:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:34:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:34:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:34:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:34:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:34:18.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:34:18.205 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.205 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.205 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.205 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.205 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:18.205 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.453359370999998, 2024-04-24T14:34:18.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:34:18.205 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.205 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.205 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.205 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.206 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.206 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:34:18.206 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.206 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.206 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.206 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.206 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:18.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.206 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.206 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.206 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.206 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:18.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:18.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:18.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:34:18.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:34:18.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:34:18.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:34:18.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:34:18.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:34:18.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:34:18.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:34:18.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:429 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3538944~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:34:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.439077931, 2024-04-24T14:34:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071866789999999996, 2024-04-24T14:34:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 429 2024-04-24T14:34:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:34:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710448+0000", 2024-04-24T14:34:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710447+0000", 2024-04-24T14:34:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710451+0000", 2024-04-24T14:34:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5930000000000002e-06 2024-04-24T14:34:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710465+0000", 2024-04-24T14:34:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.411e-05 2024-04-24T14:34:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.710470+0000", 2024-04-24T14:34:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3129999999999998e-06 2024-04-24T14:34:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716696+0000", 2024-04-24T14:34:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.006225983 2024-04-24T14:34:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.716901+0000", 2024-04-24T14:34:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020550200000000001 2024-04-24T14:34:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717634+0000", 2024-04-24T14:34:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073292099999999998 2024-04-24T14:34:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:430 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:34:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.437713623, 2024-04-24T14:34:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0068326970000000004, 2024-04-24T14:34:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 430 2024-04-24T14:34:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:34:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711812+0000", 2024-04-24T14:34:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711810+0000", 2024-04-24T14:34:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711822+0000", 2024-04-24T14:34:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2166e-05 2024-04-24T14:34:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711856+0000", 2024-04-24T14:34:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3816000000000003e-05 2024-04-24T14:34:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711873+0000", 2024-04-24T14:34:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6742999999999999e-05 2024-04-24T14:34:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717693+0000", 2024-04-24T14:34:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058204579999999997 2024-04-24T14:34:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.717864+0000", 2024-04-24T14:34:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017128299999999999 2024-04-24T14:34:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718645+0000", 2024-04-24T14:34:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078033699999999998 2024-04-24T14:34:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.437638907, 2024-04-24T14:34:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:34:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:34:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:34:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:34:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:34:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:34:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:34:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:34:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:34:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:34:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:34:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:34:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:34:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:34:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:34:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.437606975000001, 2024-04-24T14:34:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:34:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:34:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:34:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:34:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:34:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:34:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:34:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:34:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:34:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.437539459, 2024-04-24T14:34:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:34:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:34:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:34:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:34:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:34:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:34:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:34:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:34:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.380126576000002, 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:34:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:34:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:34:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:34:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:34:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:34:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:34:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:34:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:34:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:18.235 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:18.237 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:18.237 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:18.238 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:18.238 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:18.241+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:18.244 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:18.265 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:18.286 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:18.306 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:18.308 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:18.308 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:18.308 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:18.308 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:18.308 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:18.309 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:18.338 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:18.341+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:18.346 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:18.346 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:18.355 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:18.439 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:18.443+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:18.502 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:18.502 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:18.512 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:18.540 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:18.547+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:18.613 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:18.613 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:18.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:18.666 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:18.671+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:18.726 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:18.726 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:18.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:18.741 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:18.743+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:18.837 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:18.837 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:18.842 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:18.845+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:18.843 INFO:tasks.workunit.client.0.smithi173.stderr: Creating snap: 10% complete... Creating snap: 20% complete... Creating snap: 100% complete...done. 2024-04-24T14:34:18.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:18.854 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T14:34:18.854 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:18.855 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:18.855 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:18.856 INFO:tasks.workunit.client.0.smithi173.stderr:++ which dd 2024-04-24T14:34:18.858 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/dd 2024-04-24T14:34:18.858 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:18.858 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.waDYLkcsIV/data of=/dev/nbd0 bs=1M count=1 oflag=direct 2024-04-24T14:34:18.889 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records in 2024-04-24T14:34:18.889 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records out 2024-04-24T14:34:18.889 INFO:tasks.workunit.client.0.smithi173.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204947 s, 51.2 MB/s 2024-04-24T14:34:18.890 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device /dev/nbd0 18955 2024-04-24T14:34:18.890 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=/dev/nbd0 2024-04-24T14:34:18.891 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=18955 2024-04-24T14:34:18.891 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:34:18.891 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:18.891 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:18.892 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:18.893 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:18.895 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:18.895 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:18.895 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:34:18.945 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:18.949+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:18.951 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:18.952 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:18.961 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:19.044 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:19.049+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:19.055 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:19.055 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:19.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:19.145 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:19.147+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:19.158 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:19.158 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:19.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:19.245 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:19.249+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:19.262 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:19.262 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:19.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:19.346 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:19.351+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:19.370 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:19.371 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:19.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:19.447 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:19.453+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:19.480 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:19.480 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:19.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:19.548 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:19.553+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:19.590 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:19.590 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:19.599 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:19.649 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:19.653+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:19.707 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:19.707 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:19.717 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:19.750 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:19.755+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:19.820 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:19.820 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:19.829 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:19.850 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:19.855+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:19.926 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:19.927 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:19.936 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:19.951 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:19.957+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:20.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T14:34:20.032 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:20.032 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:20.041 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:20.054 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:20.059+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:20.154 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:20.157+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:20.182 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:20.182 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:20.192 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:20.255 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:20.261+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:20.291 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:20.291 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:20.300 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:20.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:34:20.355 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:20.361+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:20.446 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:20.447 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:20.454 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:20.456 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:20.456 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:20.459+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:20.474 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:20.502 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:20.525 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:20.549 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:20.554 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:20.555 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:20.557 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:20.559+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:20.564 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:20.571 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:20.574 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:20.574 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:20.574 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:20.574 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:20.575 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:20.575 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:20.658 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:20.663+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:20.660 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:20.660 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:20.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:20.660 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:20.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.660 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:34:20.661 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:20.661 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 40.593493514999999, 2024-04-24T14:34:20.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:34:20.661 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.661 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:20.661 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:20.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:20.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:34:20.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:20.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:34:20.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:34:20.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:34:20.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:34:20.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:34:20.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:34:20.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:34:20.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:34:20.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:34:20.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:34:20.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:20.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:34:20.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:34:20.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:34:20.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:34:20.665 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.665 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.665 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.665 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:34:20.665 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:20.665 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.598678935999999, 2024-04-24T14:34:20.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:34:20.665 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.665 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:20.666 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:20.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:20.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:34:20.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:20.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:34:20.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:34:20.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:34:20.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:34:20.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:34:20.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:34:20.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:34:20.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:34:20.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:34:20.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:34:20.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:20.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:34:20.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:34:20.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:34:20.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:34:20.669 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.669 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.669 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.670 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:34:20.670 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:20.670 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.587749678000002, 2024-04-24T14:34:20.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:34:20.670 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.670 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:20.670 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:20.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:20.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:34:20.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:20.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:34:20.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:34:20.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:34:20.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:34:20.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:34:20.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:34:20.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:34:20.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:34:20.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:34:20.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:34:20.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:20.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:34:20.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:34:20.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:34:20.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:34:20.674 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.674 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.674 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.674 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:34:20.674 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:20.674 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.576793596999998, 2024-04-24T14:34:20.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:34:20.675 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.675 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:20.675 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:20.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:20.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:34:20.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:20.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:34:20.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:34:20.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:34:20.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:34:20.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:34:20.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:34:20.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:34:20.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:34:20.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:34:20.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:34:20.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:20.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:34:20.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:34:20.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:34:20.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:34:20.678 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.678 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.679 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.679 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:34:20.679 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:20.679 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.565224231999998, 2024-04-24T14:34:20.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:34:20.679 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.679 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:20.679 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:20.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:20.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:34:20.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:20.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:34:20.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:34:20.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:34:20.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:34:20.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:34:20.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:34:20.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:34:20.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:34:20.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:34:20.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:34:20.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:20.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:34:20.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:34:20.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.683 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:34:20.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:34:20.683 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.683 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.683 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.683 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:34:20.683 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.312104572999999, 2024-04-24T14:34:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:34:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:20.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:34:20.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:20.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:34:20.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:34:20.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:34:20.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:34:20.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:34:20.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:34:20.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:34:20.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:34:20.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:34:20.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:34:20.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:20.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:34:20.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:34:20.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:34:20.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:34:20.688 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.688 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.688 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.688 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:34:20.688 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:20.688 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.301359404999999, 2024-04-24T14:34:20.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:34:20.688 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.688 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:20.689 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.689 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.689 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:20.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.689 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.689 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:20.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.689 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:34:20.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:20.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:34:20.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:34:20.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:34:20.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:34:20.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:34:20.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:34:20.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:34:20.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:34:20.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.692 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.693 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:34:20.693 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:20.693 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.288993129000001, 2024-04-24T14:34:20.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:34:20.693 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.693 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:20.693 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:20.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:20.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:34:20.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:20.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.695 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.695 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:34:20.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:34:20.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.695 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.695 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:34:20.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:34:20.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.695 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.695 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:34:20.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:34:20.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:34:20.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:34:20.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:34:20.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:34:20.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:20.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:34:20.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:34:20.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:34:20.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:34:20.697 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.697 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.697 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.698 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:34:20.698 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:20.698 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.164057008, 2024-04-24T14:34:20.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:34:20.698 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.698 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:20.698 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.698 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:20.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:20.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:34:20.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:20.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:34:20.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:34:20.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.700 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.700 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:34:20.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:34:20.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.700 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.700 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:34:20.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:34:20.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:34:20.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:34:20.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:34:20.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:34:20.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:20.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:34:20.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:34:20.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.702 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.702 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:34:20.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:34:20.702 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.702 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.702 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.702 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:34:20.702 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:20.702 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.144669165, 2024-04-24T14:34:20.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:34:20.702 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.704 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:34:20.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:34:20.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:34:20.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:34:20.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:34:20.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:34:20.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:20.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:34:20.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:34:20.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:34:20.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:34:20.706 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.706 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.706 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.707 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:34:20.707 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:20.707 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.134941322, 2024-04-24T14:34:20.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:34:20.707 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.707 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:20.707 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:20.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:20.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:34:20.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:20.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.708 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:34:20.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:34:20.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:34:20.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:34:20.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:34:20.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:34:20.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:34:20.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:34:20.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:34:20.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:34:20.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:20.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:34:20.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:34:20.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:34:20.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:34:20.711 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.711 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.711 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.711 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:34:20.711 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:20.711 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 25.974831642000002, 2024-04-24T14:34:20.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:34:20.711 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.711 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:20.712 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:20.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:20.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:34:20.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:20.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.713 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:34:20.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:34:20.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.713 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:34:20.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:34:20.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.713 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.714 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:34:20.714 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:34:20.714 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.714 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.714 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.714 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:34:20.714 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:34:20.714 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.714 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.714 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.714 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:34:20.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:34:20.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:20.715 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:34:20.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:34:20.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.715 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:34:20.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:34:20.715 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.715 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.716 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.716 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:34:20.716 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:20.716 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 25.965102645999998, 2024-04-24T14:34:20.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:34:20.716 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.716 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:20.716 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:20.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:20.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:34:20.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:20.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.718 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:34:20.718 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:34:20.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.718 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:34:20.718 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:34:20.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.718 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:34:20.719 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:34:20.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.719 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.719 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.719 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:34:20.719 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:34:20.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.719 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.719 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.719 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:34:20.719 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:34:20.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:34:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:34:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:34:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:34:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.720 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.630131810999998, 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.721 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.722 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.724 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:34:20.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:34:20.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.724 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.724 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:20.724 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:34:20.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:34:20.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.724 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.724 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.724 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:34:20.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:34:20.724 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.785981063000001, 2024-04-24T14:34:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:34:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.725 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:34:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:34:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:34:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:34:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:34:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:34:20.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.728 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.728 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.728 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:34:20.728 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:34:20.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.728 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.728 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.728 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:34:20.728 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:34:20.728 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.728 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.728 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.729 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.729 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:5 3.2 3.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:34:20.729 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:20.729 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 7.457743421, 2024-04-24T14:34:20.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043717790000000001, 2024-04-24T14:34:20.729 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.729 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-24T14:34:20.729 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:20.729 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:34:20.729 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:34:20.729 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 5 2024-04-24T14:34:20.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.729 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207678+0000", 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999962 2024-04-24T14:34:20.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.731 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.731 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207700+0000", 2024-04-24T14:34:20.731 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1313000000000001e-05 2024-04-24T14:34:20.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.731 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.731 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207757+0000", 2024-04-24T14:34:20.731 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7499e-05 2024-04-24T14:34:20.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.731 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.731 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207779+0000", 2024-04-24T14:34:20.731 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1478999999999999e-05 2024-04-24T14:34:20.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.212054+0000", 2024-04-24T14:34:20.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042750849999999996 2024-04-24T14:34:20.732 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.732 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.732 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.732 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:20.732 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:20.732 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 2.562218063, 2024-04-24T14:34:20.733 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044084859999999997, 2024-04-24T14:34:20.733 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.733 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:20.733 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.733 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.733 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:20.733 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.733 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.733 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:20.733 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.734 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103206+0000", 2024-04-24T14:34:20.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:20.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.734 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104731+0000", 2024-04-24T14:34:20.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001524755 2024-04-24T14:34:20.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104759+0000", 2024-04-24T14:34:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7973000000000001e-05 2024-04-24T14:34:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104807+0000", 2024-04-24T14:34:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8112000000000003e-05 2024-04-24T14:34:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104907+0000", 2024-04-24T14:34:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100317 2024-04-24T14:34:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.736 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.736 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.736 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104918+0000", 2024-04-24T14:34:20.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747999999999999e-05 2024-04-24T14:34:20.736 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.736 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.736 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.736 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.107616+0000", 2024-04-24T14:34:20.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026977559999999999 2024-04-24T14:34:20.736 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.736 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.736 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.736 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.736 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.737 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(3.2 83/82 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:20.737 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:20.737 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 2.5558236860000001, 2024-04-24T14:34:20.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041425760000000002, 2024-04-24T14:34:20.737 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.737 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:20.737 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.737 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.737 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:20.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:20.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109600+0000", 2024-04-24T14:34:20.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:20.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111210+0000", 2024-04-24T14:34:20.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016097279999999999 2024-04-24T14:34:20.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.739 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.739 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111237+0000", 2024-04-24T14:34:20.739 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7189999999999999e-05 2024-04-24T14:34:20.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.739 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.739 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111251+0000", 2024-04-24T14:34:20.739 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.365e-05 2024-04-24T14:34:20.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.739 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.739 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111367+0000", 2024-04-24T14:34:20.739 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000115703 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111378+0000", 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1873e-05 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.113744+0000", 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023657069999999999 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.740 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 2.5389231099999998, 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046193859999999996, 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.741 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:20.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.742 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126501+0000", 2024-04-24T14:34:20.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:20.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.742 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128169+0000", 2024-04-24T14:34:20.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016686229999999999 2024-04-24T14:34:20.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128202+0000", 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2539000000000001e-05 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128222+0000", 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0149e-05 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128375+0000", 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000153459 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.744 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.744 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128399+0000", 2024-04-24T14:34:20.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3393000000000001e-05 2024-04-24T14:34:20.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.744 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.744 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.131122+0000", 2024-04-24T14:34:20.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027227140000000002 2024-04-24T14:34:20.744 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.744 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.744 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.744 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 49'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@49'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:20.745 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:20.745 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 2.5171417300000001, 2024-04-24T14:34:20.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038414529999999999, 2024-04-24T14:34:20.745 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:20.745 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:20.745 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:20.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:20.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:20.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:20.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:20.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:20.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:20.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148282+0000", 2024-04-24T14:34:20.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:20.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.746 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:20.746 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149875+0000", 2024-04-24T14:34:20.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015929900000000001 2024-04-24T14:34:20.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:20.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149908+0000", 2024-04-24T14:34:20.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3102000000000003e-05 2024-04-24T14:34:20.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:20.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149941+0000", 2024-04-24T14:34:20.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2480000000000001e-05 2024-04-24T14:34:20.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:20.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150031+0000", 2024-04-24T14:34:20.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0394000000000002e-05 2024-04-24T14:34:20.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:20.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150043+0000", 2024-04-24T14:34:20.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1967000000000001e-05 2024-04-24T14:34:20.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:20.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:20.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:20.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.152125+0000", 2024-04-24T14:34:20.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020817029999999999 2024-04-24T14:34:20.748 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.748 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.748 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.749 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:20.749 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:20.749 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:20.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:20.758 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:20.761+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:20.859 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:20.860 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:20.860 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:20.860 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:20.860 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:20.863+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:20.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:20.961 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:20.963+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:21.010 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-24T14:34:21.019 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:21.019 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:21.019 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:21.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:21.074 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:21.079+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:21.124 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.247570463999999, 2024-04-24T14:34:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:34:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.124 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:34:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:34:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:34:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:34:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:34:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:34:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:34:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:34:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:34:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:34:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:34:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:34:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:34:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:34:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:34:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:34:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:34:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:34:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:34:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:34:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:34:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:34:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.247541478999999, 2024-04-24T14:34:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:34:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:34:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:34:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:34:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:34:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:34:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:34:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:34:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:34:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:34:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:34:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:34:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:34:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:34:21.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:34:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:34:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:34:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:34:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:34:21.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:34:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:34:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:34:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:21.136 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.247522994000001, 2024-04-24T14:34:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:34:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:34:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:34:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:34:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:34:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:34:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:34:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:34:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:34:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:34:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:34:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:34:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:34:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:34:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:34:21.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:34:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:34:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:34:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:34:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:34:21.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:34:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:34:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:34:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:21.142 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.247507713000001, 2024-04-24T14:34:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:34:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:34:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:21.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:34:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:34:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:34:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:34:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:34:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:34:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:34:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:34:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:34:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:34:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:34:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:34:21.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:34:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:34:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:34:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:34:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:34:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:34:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:34:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:34:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:21.148 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.247491931999999, 2024-04-24T14:34:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:34:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:34:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:34:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:34:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:34:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:34:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:34:21.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:34:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:34:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:34:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:34:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:34:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:34:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:34:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:34:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:34:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:34:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:34:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:34:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:34:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:34:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:34:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.154 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.247477644, 2024-04-24T14:34:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:34:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:34:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:34:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:34:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:34:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:34:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:34:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:34:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:34:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:34:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:34:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:34:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:34:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:34:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:34:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:34:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:34:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:34:21.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:34:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:34:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:34:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:34:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.247457892, 2024-04-24T14:34:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:34:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:34:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:34:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:34:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:34:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:34:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:34:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:34:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:34:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:34:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:34:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:34:21.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:34:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:34:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:34:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:34:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:34:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:34:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:34:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:34:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:34:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:34:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.166 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.247440444999999, 2024-04-24T14:34:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:34:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:21.167 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:34:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:34:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:34:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:34:21.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:34:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:34:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:34:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:34:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:34:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:34:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:34:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:34:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:34:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:34:21.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:34:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:34:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:34:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:34:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:34:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:34:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:34:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4d9853ae:::rbd_id.INVALIDIMAGE:head, version: 53'12, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4d9853ae:::rbd_id.INVALIDIMAGE:head@53'12, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.222161914000001, 2024-04-24T14:34:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075925020000000001, 2024-04-24T14:34:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.173 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905243+0000", 2024-04-24T14:34:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905253+0000", 2024-04-24T14:34:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1619999999999997e-06 2024-04-24T14:34:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905262+0000", 2024-04-24T14:34:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2399999999999996e-06 2024-04-24T14:34:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905268+0000", 2024-04-24T14:34:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2530000000000001e-06 2024-04-24T14:34:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912457+0000", 2024-04-24T14:34:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071890779999999998 2024-04-24T14:34:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912462+0000", 2024-04-24T14:34:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5159999999999996e-06 2024-04-24T14:34:21.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912836+0000", 2024-04-24T14:34:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037484699999999998 2024-04-24T14:34:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 62'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@62'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.198542985, 2024-04-24T14:34:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036829689999999999, 2024-04-24T14:34:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928862+0000", 2024-04-24T14:34:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930264+0000", 2024-04-24T14:34:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014016059999999999 2024-04-24T14:34:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930281+0000", 2024-04-24T14:34:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7859999999999998e-05 2024-04-24T14:34:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930291+0000", 2024-04-24T14:34:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9299999999999998e-06 2024-04-24T14:34:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930894+0000", 2024-04-24T14:34:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060250899999999999 2024-04-24T14:34:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930903+0000", 2024-04-24T14:34:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4599999999999992e-06 2024-04-24T14:34:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.932546+0000", 2024-04-24T14:34:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016425019999999999 2024-04-24T14:34:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head, version: 62'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head@62'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.192828037000002, 2024-04-24T14:34:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037783690000000002, 2024-04-24T14:34:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934576+0000", 2024-04-24T14:34:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936033+0000", 2024-04-24T14:34:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001456658 2024-04-24T14:34:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936071+0000", 2024-04-24T14:34:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7752e-05 2024-04-24T14:34:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936087+0000", 2024-04-24T14:34:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5919000000000001e-05 2024-04-24T14:34:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936226+0000", 2024-04-24T14:34:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013910800000000001 2024-04-24T14:34:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936240+0000", 2024-04-24T14:34:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.456e-05 2024-04-24T14:34:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938356+0000", 2024-04-24T14:34:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021157540000000001 2024-04-24T14:34:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 62'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@62'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.191781508999998, 2024-04-24T14:34:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052678259999999998, 2024-04-24T14:34:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935623+0000", 2024-04-24T14:34:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937055+0000", 2024-04-24T14:34:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014314480000000001 2024-04-24T14:34:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937085+0000", 2024-04-24T14:34:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0204000000000001e-05 2024-04-24T14:34:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937097+0000", 2024-04-24T14:34:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1708e-05 2024-04-24T14:34:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938410+0000", 2024-04-24T14:34:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013128910000000001 2024-04-24T14:34:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938419+0000", 2024-04-24T14:34:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1740000000000006e-06 2024-04-24T14:34:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.940892+0000", 2024-04-24T14:34:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024734990000000001 2024-04-24T14:34:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 62'82, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@62'82, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.182270118000002, 2024-04-24T14:34:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003877447, 2024-04-24T14:34:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945134+0000", 2024-04-24T14:34:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946323+0000", 2024-04-24T14:34:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001188635 2024-04-24T14:34:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946352+0000", 2024-04-24T14:34:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9238e-05 2024-04-24T14:34:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946374+0000", 2024-04-24T14:34:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1396999999999999e-05 2024-04-24T14:34:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.947098+0000", 2024-04-24T14:34:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072458199999999998 2024-04-24T14:34:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.947109+0000", 2024-04-24T14:34:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0557000000000001e-05 2024-04-24T14:34:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.949013+0000", 2024-04-24T14:34:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019044929999999999 2024-04-24T14:34:21.193 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.175813132999998, 2024-04-24T14:34:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041579369999999996, 2024-04-24T14:34:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951585+0000", 2024-04-24T14:34:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999928 2024-04-24T14:34:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953339+0000", 2024-04-24T14:34:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017540019999999999 2024-04-24T14:34:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953363+0000", 2024-04-24T14:34:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3178000000000001e-05 2024-04-24T14:34:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953374+0000", 2024-04-24T14:34:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1015e-05 2024-04-24T14:34:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953522+0000", 2024-04-24T14:34:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014857800000000001 2024-04-24T14:34:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953565+0000", 2024-04-24T14:34:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2511e-05 2024-04-24T14:34:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955751+0000", 2024-04-24T14:34:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002185958 2024-04-24T14:34:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 62'82, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@62'82, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.175398616999999, 2024-04-24T14:34:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005349749, 2024-04-24T14:34:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952006+0000", 2024-04-24T14:34:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953689+0000", 2024-04-24T14:34:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00168326 2024-04-24T14:34:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953717+0000", 2024-04-24T14:34:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7994e-05 2024-04-24T14:34:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953746+0000", 2024-04-24T14:34:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8059999999999999e-05 2024-04-24T14:34:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955798+0000", 2024-04-24T14:34:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002052089 2024-04-24T14:34:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955812+0000", 2024-04-24T14:34:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2024-04-24T14:34:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.957357+0000", 2024-04-24T14:34:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001545537 2024-04-24T14:34:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.166608498999999, 2024-04-24T14:34:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051567999999999996, 2024-04-24T14:34:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960796+0000", 2024-04-24T14:34:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962459+0000", 2024-04-24T14:34:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016622259999999999 2024-04-24T14:34:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962481+0000", 2024-04-24T14:34:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2359999999999999e-05 2024-04-24T14:34:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962496+0000", 2024-04-24T14:34:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.523e-05 2024-04-24T14:34:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.964299+0000", 2024-04-24T14:34:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018025129999999999 2024-04-24T14:34:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.964316+0000", 2024-04-24T14:34:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7436999999999999e-05 2024-04-24T14:34:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.965954+0000", 2024-04-24T14:34:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016381099999999999 2024-04-24T14:34:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head, version: 62'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head@62'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.133602310000001, 2024-04-24T14:34:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003994821, 2024-04-24T14:34:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993802+0000", 2024-04-24T14:34:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994846+0000", 2024-04-24T14:34:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00104368 2024-04-24T14:34:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994884+0000", 2024-04-24T14:34:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7796000000000002e-05 2024-04-24T14:34:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994900+0000", 2024-04-24T14:34:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5917e-05 2024-04-24T14:34:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995046+0000", 2024-04-24T14:34:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014653499999999999 2024-04-24T14:34:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995055+0000", 2024-04-24T14:34:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0759999999999992e-06 2024-04-24T14:34:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.997798+0000", 2024-04-24T14:34:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027432709999999998 2024-04-24T14:34:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(2.2 68/67 [PushOp(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 49'1, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@49'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.133294851999999, 2024-04-24T14:34:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038339310000000001, 2024-04-24T14:34:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994110+0000", 2024-04-24T14:34:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994914+0000", 2024-04-24T14:34:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080383099999999997 2024-04-24T14:34:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994943+0000", 2024-04-24T14:34:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9475000000000002e-05 2024-04-24T14:34:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994955+0000", 2024-04-24T14:34:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1525000000000001e-05 2024-04-24T14:34:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995123+0000", 2024-04-24T14:34:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016812899999999999 2024-04-24T14:34:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995143+0000", 2024-04-24T14:34:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9724999999999998e-05 2024-04-24T14:34:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.997945+0000", 2024-04-24T14:34:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028023890000000002 2024-04-24T14:34:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head, version: 62'144, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head@62'144, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.092989657, 2024-04-24T14:34:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041651270000000002, 2024-04-24T14:34:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034415+0000", 2024-04-24T14:34:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035025+0000", 2024-04-24T14:34:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060999700000000003 2024-04-24T14:34:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035077+0000", 2024-04-24T14:34:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2132999999999997e-05 2024-04-24T14:34:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035100+0000", 2024-04-24T14:34:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2690000000000001e-05 2024-04-24T14:34:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035293+0000", 2024-04-24T14:34:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019358900000000001 2024-04-24T14:34:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035311+0000", 2024-04-24T14:34:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7952999999999999e-05 2024-04-24T14:34:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.038581+0000", 2024-04-24T14:34:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032700670000000002 2024-04-24T14:34:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head, version: 62'144, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head@62'144, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:21.219 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.078824222000001, 2024-04-24T14:34:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039835970000000002, 2024-04-24T14:34:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048581+0000", 2024-04-24T14:34:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049792+0000", 2024-04-24T14:34:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012111579999999999 2024-04-24T14:34:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049823+0000", 2024-04-24T14:34:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1137999999999999e-05 2024-04-24T14:34:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049838+0000", 2024-04-24T14:34:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5061e-05 2024-04-24T14:34:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.050005+0000", 2024-04-24T14:34:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016745400000000001 2024-04-24T14:34:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.050022+0000", 2024-04-24T14:34:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7022000000000001e-05 2024-04-24T14:34:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.052565+0000", 2024-04-24T14:34:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025428669999999999 2024-04-24T14:34:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.223 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:21.225 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:21.226 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:21.229+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:21.326 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:21.331+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:21.334 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:21.334 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:21.334 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:21.334 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:21.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:21.427 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:21.431+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:21.488 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:21.488 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:21.488 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:21.488 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:21.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:21.528 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:21.533+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:21.603 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:97 3.1 3:990e5750:::rbd_header.12ff64619e2d:head [watch reconnect cookie 140276853203408 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.253759254000002, 2024-04-24T14:34:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012161632, 2024-04-24T14:34:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 97 2024-04-24T14:34:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352830+0000", 2024-04-24T14:34:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352834+0000", 2024-04-24T14:34:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2699999999999998e-06 2024-04-24T14:34:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352844+0000", 2024-04-24T14:34:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0406e-05 2024-04-24T14:34:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352850+0000", 2024-04-24T14:34:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6439999999999997e-06 2024-04-24T14:34:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362391+0000", 2024-04-24T14:34:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095408969999999996 2024-04-24T14:34:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362683+0000", 2024-04-24T14:34:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029238000000000002 2024-04-24T14:34:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.364992+0000", 2024-04-24T14:34:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023092020000000002 2024-04-24T14:34:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:225 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.101317141999999, 2024-04-24T14:34:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010959015000000001, 2024-04-24T14:34:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 225 2024-04-24T14:34:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505275+0000", 2024-04-24T14:34:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0269999999999998e-06 2024-04-24T14:34:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505279+0000", 2024-04-24T14:34:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4279999999999998e-06 2024-04-24T14:34:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505282+0000", 2024-04-24T14:34:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.982e-06 2024-04-24T14:34:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515209+0000", 2024-04-24T14:34:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099266419999999994 2024-04-24T14:34:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515466+0000", 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025719700000000002 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.516232+0000", 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076608899999999996 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.850285399999997, 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:34:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:34:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:34:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:34:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:34:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:34:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:34:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:34:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:34:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:34:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:34:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:34:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:34:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:34:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:34:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:34:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:34:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:34:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:34:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:34:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:34:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.850234565999997, 2024-04-24T14:34:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:34:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:34:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:34:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:34:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:34:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:34:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:34:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:34:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:34:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:34:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:34:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:34:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:34:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:34:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:34:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:34:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:34:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:34:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:34:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:34:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:34:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:34:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.850198259000003, 2024-04-24T14:34:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:34:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:34:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:34:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:34:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:34:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:34:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:34:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:34:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:34:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:34:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:34:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:34:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:34:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:34:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:34:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:34:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:34:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.846787009000003, 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:34:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:34:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:34:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:34:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:34:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:34:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:34:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:34:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:34:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:34:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:34:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:34:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:34:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:34:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:34:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:34:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:34:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:34:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:34:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:34:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:34:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.846725331999998, 2024-04-24T14:34:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:34:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:34:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:34:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:34:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:34:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:34:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:34:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:34:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:34:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:34:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:34:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:34:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:34:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:34:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.846690756000001, 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:34:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:34:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:34:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:34:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:34:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:34:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:34:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:34:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:34:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:34:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:34:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:34:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:34:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:34:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.846661417, 2024-04-24T14:34:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:34:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:34:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:34:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:34:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:34:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:34:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:34:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:34:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:34:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:34:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:34:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:34:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:34:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:34:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:34:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:34:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:34:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.846584835000002, 2024-04-24T14:34:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:34:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:34:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:34:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:34:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:34:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:34:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:34:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:34:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:34:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:34:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:34:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:34:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:34:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:34:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:34:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:34:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:34:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:34:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.846537290000001, 2024-04-24T14:34:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:34:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:34:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:34:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:34:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:34:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:34:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:34:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:34:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:34:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:34:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:34:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:34:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:34:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:34:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:34:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:34:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:34:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:34:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:34:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:34:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:34:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:34:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:34:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.846496647999999, 2024-04-24T14:34:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:34:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:34:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:34:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:34:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:34:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:34:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:34:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:34:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:34:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:34:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:34:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:34:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:34:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:34:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:34:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:34:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.846479537, 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:34:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:34:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:34:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:34:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:34:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:34:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:34:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:34:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:34:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:34:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:34:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:34:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:34:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:34:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:34:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:34:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:34:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:34:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:34:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:34:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:34:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:481 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.812607829999997, 2024-04-24T14:34:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010661718000000001, 2024-04-24T14:34:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 481 2024-04-24T14:34:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793981+0000", 2024-04-24T14:34:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793985+0000", 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8469999999999998e-06 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0220000000000004e-06 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793998+0000", 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8559999999999998e-06 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803807+0000", 2024-04-24T14:34:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098088629999999993 2024-04-24T14:34:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803916+0000", 2024-04-24T14:34:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010854899999999999 2024-04-24T14:34:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.804644+0000", 2024-04-24T14:34:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072835399999999996 2024-04-24T14:34:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:497 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.794667517999997, 2024-04-24T14:34:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013996471, 2024-04-24T14:34:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 497 2024-04-24T14:34:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811922+0000", 2024-04-24T14:34:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811925+0000", 2024-04-24T14:34:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4120000000000001e-06 2024-04-24T14:34:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811936+0000", 2024-04-24T14:34:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1077000000000001e-05 2024-04-24T14:34:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811941+0000", 2024-04-24T14:34:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7539999999999997e-06 2024-04-24T14:34:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822289+0000", 2024-04-24T14:34:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010347635000000001 2024-04-24T14:34:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822457+0000", 2024-04-24T14:34:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000168257 2024-04-24T14:34:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.825919+0000", 2024-04-24T14:34:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003461979 2024-04-24T14:34:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:25 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.639109576999999, 2024-04-24T14:34:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013349789000000001, 2024-04-24T14:34:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 25 2024-04-24T14:34:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967479+0000", 2024-04-24T14:34:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967669+0000", 2024-04-24T14:34:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000189829 2024-04-24T14:34:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967687+0000", 2024-04-24T14:34:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8199999999999999e-05 2024-04-24T14:34:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967695+0000", 2024-04-24T14:34:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4869999999999996e-06 2024-04-24T14:34:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.972501+0000", 2024-04-24T14:34:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048057509999999996 2024-04-24T14:34:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.972824+0000", 2024-04-24T14:34:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032322299999999998 2024-04-24T14:34:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.973071+0000", 2024-04-24T14:34:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024751299999999999 2024-04-24T14:34:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977624+0000", 2024-04-24T14:34:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045527570000000002 2024-04-24T14:34:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980627+0000", 2024-04-24T14:34:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003002972 2024-04-24T14:34:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980743+0000", 2024-04-24T14:34:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2024-04-24T14:34:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980830+0000", 2024-04-24T14:34:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7534999999999995e-05 2024-04-24T14:34:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:26 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.638885842000001, 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013403597999999999, 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 26 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967703+0000", 2024-04-24T14:34:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967874+0000", 2024-04-24T14:34:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017088499999999999 2024-04-24T14:34:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967886+0000", 2024-04-24T14:34:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1744e-05 2024-04-24T14:34:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967893+0000", 2024-04-24T14:34:21.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2024-04-24T14:34:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974028+0000", 2024-04-24T14:34:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061356650000000002 2024-04-24T14:34:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974347+0000", 2024-04-24T14:34:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031897400000000003 2024-04-24T14:34:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:21.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974774+0000", 2024-04-24T14:34:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042661700000000001 2024-04-24T14:34:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980930+0000", 2024-04-24T14:34:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061563360000000001 2024-04-24T14:34:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980945+0000", 2024-04-24T14:34:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4864e-05 2024-04-24T14:34:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981035+0000", 2024-04-24T14:34:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9390999999999998e-05 2024-04-24T14:34:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981108+0000", 2024-04-24T14:34:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3255000000000003e-05 2024-04-24T14:34:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:27 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.638688695999999, 2024-04-24T14:34:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013394262000000001, 2024-04-24T14:34:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 27 2024-04-24T14:34:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967900+0000", 2024-04-24T14:34:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968061+0000", 2024-04-24T14:34:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016101900000000001 2024-04-24T14:34:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968072+0000", 2024-04-24T14:34:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1012999999999999e-05 2024-04-24T14:34:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968079+0000", 2024-04-24T14:34:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.46e-06 2024-04-24T14:34:21.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.976820+0000", 2024-04-24T14:34:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087406289999999998 2024-04-24T14:34:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.976951+0000", 2024-04-24T14:34:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131187 2024-04-24T14:34:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977129+0000", 2024-04-24T14:34:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000178135 2024-04-24T14:34:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981126+0000", 2024-04-24T14:34:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00399709 2024-04-24T14:34:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981193+0000", 2024-04-24T14:34:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7083999999999997e-05 2024-04-24T14:34:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981251+0000", 2024-04-24T14:34:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7754000000000003e-05 2024-04-24T14:34:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981296+0000", 2024-04-24T14:34:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4765000000000003e-05 2024-04-24T14:34:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:28 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.638503463999999, 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013427247999999999, 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 28 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968086+0000", 2024-04-24T14:34:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968247+0000", 2024-04-24T14:34:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016121199999999999 2024-04-24T14:34:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968260+0000", 2024-04-24T14:34:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.275e-05 2024-04-24T14:34:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968266+0000", 2024-04-24T14:34:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3770000000000001e-06 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977699+0000", 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094331439999999992 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977894+0000", 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194945 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.978258+0000", 2024-04-24T14:34:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036359400000000002 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981140+0000", 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028821540000000001 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981360+0000", 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022056300000000001 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981444+0000", 2024-04-24T14:34:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3733999999999995e-05 2024-04-24T14:34:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981514+0000", 2024-04-24T14:34:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9702000000000005e-05 2024-04-24T14:34:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:29 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.638316619000001, 2024-04-24T14:34:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014492283, 2024-04-24T14:34:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 29 2024-04-24T14:34:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968434+0000", 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016130199999999999 2024-04-24T14:34:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968484+0000", 2024-04-24T14:34:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0003000000000002e-05 2024-04-24T14:34:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968499+0000", 2024-04-24T14:34:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4628e-05 2024-04-24T14:34:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979299+0000", 2024-04-24T14:34:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010800624999999999 2024-04-24T14:34:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979488+0000", 2024-04-24T14:34:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000189037 2024-04-24T14:34:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979784+0000", 2024-04-24T14:34:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029562000000000002 2024-04-24T14:34:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982305+0000", 2024-04-24T14:34:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025212339999999998 2024-04-24T14:34:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982680+0000", 2024-04-24T14:34:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000374762 2024-04-24T14:34:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982731+0000", 2024-04-24T14:34:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1146999999999997e-05 2024-04-24T14:34:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982766+0000", 2024-04-24T14:34:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4755000000000002e-05 2024-04-24T14:34:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:21.716 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:21.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:21.718 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:21.721+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:21.819 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:21.825+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:21.825 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:21.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:21.920 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:21.925+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:21.936 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:21.936 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:21.937 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:21.937 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:21.937 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:21.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:21.998 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 1] 2024-04-24T14:34:21.999 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:34:21.999 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5] 2024-04-24T14:34:21.999 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-04-24T14:34:22.021 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:22.025+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:22.052 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:22.053 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:22.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:22.053 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:22.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.053 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:192 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:22.053 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:22.053 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.585920215000002, 2024-04-24T14:34:22.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010036329, 2024-04-24T14:34:22.053 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.053 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.054 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.054 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.054 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192 2024-04-24T14:34:22.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.054 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:22.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:22.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471099+0000", 2024-04-24T14:34:22.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471103+0000", 2024-04-24T14:34:22.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8800000000000001e-06 2024-04-24T14:34:22.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471111+0000", 2024-04-24T14:34:22.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2930000000000001e-06 2024-04-24T14:34:22.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471116+0000", 2024-04-24T14:34:22.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1800000000000004e-06 2024-04-24T14:34:22.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480338+0000", 2024-04-24T14:34:22.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092213069999999998 2024-04-24T14:34:22.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480467+0000", 2024-04-24T14:34:22.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012901500000000001 2024-04-24T14:34:22.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481136+0000", 2024-04-24T14:34:22.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066948400000000003 2024-04-24T14:34:22.057 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.057 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.057 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.058 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:22.058 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:22.058 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.585897179, 2024-04-24T14:34:22.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:34:22.058 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.058 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.058 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.058 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.058 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.058 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:34:22.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.058 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:22.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:22.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:34:22.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:34:22.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:34:22.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:34:22.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:34:22.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:34:22.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:34:22.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:34:22.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:34:22.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:34:22.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:34:22.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:34:22.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:34:22.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.062 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.062 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.062 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:22.062 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.210540475999998, 2024-04-24T14:34:22.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:34:22.063 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.063 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.063 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:34:22.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.063 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:22.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:22.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:34:22.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:34:22.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:34:22.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:34:22.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:34:22.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:34:22.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:22.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:34:22.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:34:22.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:34:22.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:34:22.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:34:22.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:34:22.066 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.066 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.067 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:526 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.067 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:22.067 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.209152953, 2024-04-24T14:34:22.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010568783999999999, 2024-04-24T14:34:22.067 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.067 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.067 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.067 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.067 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.067 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 526 2024-04-24T14:34:22.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.068 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:22.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:22.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847866+0000", 2024-04-24T14:34:22.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847871+0000", 2024-04-24T14:34:22.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1590000000000001e-06 2024-04-24T14:34:22.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847883+0000", 2024-04-24T14:34:22.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.172e-05 2024-04-24T14:34:22.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847890+0000", 2024-04-24T14:34:22.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6950000000000003e-06 2024-04-24T14:34:22.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857418+0000", 2024-04-24T14:34:22.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095285909999999995 2024-04-24T14:34:22.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857588+0000", 2024-04-24T14:34:22.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169541 2024-04-24T14:34:22.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858436+0000", 2024-04-24T14:34:22.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084814700000000005 2024-04-24T14:34:22.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.071 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.071 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.071 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:22.072 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.209112322000003, 2024-04-24T14:34:22.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:34:22.072 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.072 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.072 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:34:22.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.072 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:22.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:22.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:34:22.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:34:22.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:34:22.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:34:22.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:34:22.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:34:22.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:34:22.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:34:22.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:34:22.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:34:22.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:34:22.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:34:22.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:34:22.076 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.076 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.076 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.076 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.076 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:22.076 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.209085233000003, 2024-04-24T14:34:22.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:34:22.076 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.076 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.076 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.077 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.077 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.077 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:34:22.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.077 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:22.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:22.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:34:22.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:22.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:34:22.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:34:22.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:34:22.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:34:22.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:34:22.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:34:22.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:34:22.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:34:22.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:34:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:34:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:34:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:34:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.209061265999999, 2024-04-24T14:34:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:34:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:34:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:34:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:34:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:34:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:34:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:34:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:34:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:34:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:34:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:34:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:34:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:34:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:34:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:34:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.195077286999997, 2024-04-24T14:34:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:34:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:34:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:34:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:34:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:34:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:34:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:34:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:34:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:34:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:34:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:34:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:34:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:34:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:34:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:34:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:34:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:34:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:34:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:34:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:34:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:34:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:34:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:34:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.194960975999997, 2024-04-24T14:34:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:34:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:34:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:34:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:34:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:34:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:34:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:34:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:34:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:34:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:34:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:34:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:34:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:34:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:34:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:34:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:34:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:34:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:34:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:34:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:34:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:34:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:34:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:34:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.194913317999998, 2024-04-24T14:34:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:34:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:34:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:34:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:34:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:34:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:34:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:34:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:34:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:34:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:34:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:34:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:34:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:34:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:34:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:34:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:34:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:34:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:34:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:34:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:34:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:34:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:34:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:34:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.190876019999997, 2024-04-24T14:34:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:34:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:34:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:34:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:34:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:34:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:34:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:34:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:34:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:34:22.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:34:22.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:34:22.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:34:22.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:34:22.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:34:22.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:34:22.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:34:22.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:34:22.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:34:22.109 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.109 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.110 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.110 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:22.110 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.190812260999998, 2024-04-24T14:34:22.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:34:22.110 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.110 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.110 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.110 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.110 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:34:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:34:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:34:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:34:22.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:34:22.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:34:22.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:34:22.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:34:22.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:34:22.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:34:22.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:34:22.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:34:22.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:34:22.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:34:22.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:34:22.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:34:22.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:34:22.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:34:22.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:34:22.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:34:22.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:34:22.116 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.116 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.116 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.116 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.116 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:22.116 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.190786770999999, 2024-04-24T14:34:22.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:34:22.116 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.117 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.117 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:34:22.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.117 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:22.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:22.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:22.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:22.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:22.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:34:22.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:34:22.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:34:22.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:34:22.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:34:22.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:34:22.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:34:22.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:34:22.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:34:22.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:34:22.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:34:22.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:34:22.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:34:22.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:34:22.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:34:22.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:34:22.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:34:22.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:34:22.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:34:22.122 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.122 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.122 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.122 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.122 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:22.122 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.190763021000002, 2024-04-24T14:34:22.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:34:22.123 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.123 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.123 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.123 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.123 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.123 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:34:22.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.123 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:22.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:34:22.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:34:22.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:34:22.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:34:22.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:34:22.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:34:22.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:34:22.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:34:22.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:34:22.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:34:22.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:34:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:34:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:34:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:34:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:34:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:34:22.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:34:22.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:34:22.128 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.128 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.128 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.128 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.128 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:22.128 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.190724508000002, 2024-04-24T14:34:22.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:34:22.129 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.129 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.129 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.129 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.129 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.129 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:34:22.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.129 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:22.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:22.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:34:22.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:22.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:34:22.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:34:22.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:34:22.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:34:22.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:34:22.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:34:22.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:34:22.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:34:22.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:34:22.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:34:22.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:34:22.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:34:22.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:34:22.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:34:22.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:34:22.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:34:22.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:34:22.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:34:22.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:34:22.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:34:22.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.134 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.190696311000004, 2024-04-24T14:34:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:34:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:34:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:34:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:34:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:34:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:34:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:34:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:34:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:34:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:34:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:34:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:34:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:34:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:34:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:34:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:34:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:34:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:34:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:34:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:34:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:34:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:34:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:34:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.190674854000001, 2024-04-24T14:34:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:34:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:34:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.142 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:22.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:22.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:34:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:34:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:34:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:34:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:34:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:34:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:34:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:34:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:34:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:34:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:34:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:34:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:34:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:34:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:34:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:34:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:34:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:34:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:34:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:34:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.190656011999998, 2024-04-24T14:34:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:34:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:34:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:34:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:34:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:34:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:34:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:34:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:34:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:34:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:34:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:34:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:34:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:34:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:34:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:34:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:34:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:34:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:34:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:34:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:34:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:34:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:34:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:34:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:561 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.148693391999998, 2024-04-24T14:34:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01066806, 2024-04-24T14:34:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 561 2024-04-24T14:34:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908326+0000", 2024-04-24T14:34:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908329+0000", 2024-04-24T14:34:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.802e-06 2024-04-24T14:34:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908338+0000", 2024-04-24T14:34:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6260000000000001e-06 2024-04-24T14:34:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908345+0000", 2024-04-24T14:34:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1389999999999998e-06 2024-04-24T14:34:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.917978+0000", 2024-04-24T14:34:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096329840000000007 2024-04-24T14:34:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918178+0000", 2024-04-24T14:34:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019999199999999999 2024-04-24T14:34:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918995+0000", 2024-04-24T14:34:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081708099999999999 2024-04-24T14:34:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.977823863000001, 2024-04-24T14:34:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:34:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:34:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:34:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:34:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:34:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:34:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:34:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:34:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:34:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:34:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:34:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:34:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:34:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:34:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:34:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:34:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:34:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:34:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:34:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:34:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:34:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:34:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:34:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:22.164 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:22.165 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:22.165 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:22.167+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:22.266 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:22.271+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:22.272 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:22.272 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:22.272 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:22.272 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:22.281 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:22.290 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-04-24T14:34:22.295+0000 7ff9c25de700 -1 mon.a@0(leader).osd e85 definitely_dead 0 2024-04-24T14:34:22.367 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:22.373+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:22.383 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:22.384 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:22.384 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:22.393 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:22.467 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:22.473+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:22.491 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:22.491 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:22.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:22.491 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:22.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.492 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:22.492 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:22.492 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 119.86863413, 2024-04-24T14:34:22.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:34:22.492 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.492 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:22.492 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.492 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:22.492 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:22.492 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:34:22.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.492 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:22.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:22.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:34:22.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:22.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:34:22.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:34:22.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:34:22.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:34:22.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:34:22.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:34:22.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:34:22.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:34:22.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:34:22.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:34:22.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:22.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:34:22.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:34:22.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:22.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:34:22.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:34:22.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:34:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:34:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:34:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:34:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:34:22.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:34:22.498 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.498 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.498 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.498 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:22.498 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:22.498 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 119.86845704300001, 2024-04-24T14:34:22.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:34:22.498 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.498 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:22.498 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.498 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:22.499 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:22.499 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:34:22.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.499 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:22.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:22.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:34:22.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:34:22.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:34:22.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:34:22.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:34:22.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:34:22.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:34:22.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:22.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:34:22.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:34:22.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:22.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:34:22.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:34:22.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:22.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:34:22.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:34:22.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:22.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:34:22.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:34:22.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:34:22.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:34:22.503 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.503 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.503 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.504 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:22.504 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:22.504 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 119.868353129, 2024-04-24T14:34:22.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:34:22.504 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.504 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:22.504 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.504 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:22.504 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:22.504 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:34:22.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.505 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:22.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:22.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:34:22.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:22.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:34:22.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:34:22.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:34:22.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:34:22.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:34:22.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:22.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:34:22.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:34:22.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:34:22.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:34:22.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:22.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:34:22.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:34:22.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:22.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:34:22.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:34:22.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:22.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:34:22.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:34:22.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:22.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:34:22.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:34:22.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:34:22.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:34:22.509 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.509 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.509 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.510 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:22.510 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:22.510 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 119.86796467800001, 2024-04-24T14:34:22.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:34:22.510 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.510 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:22.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:22.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:22.510 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:34:22.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.510 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:22.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:22.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:34:22.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:22.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:34:22.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:34:22.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:34:22.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:34:22.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:34:22.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:34:22.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:34:22.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:22.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:34:22.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:34:22.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:22.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:34:22.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:34:22.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:22.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:34:22.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:34:22.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:34:22.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:34:22.515 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.515 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.515 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.515 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:22.515 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:22.515 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 119.86787946600001, 2024-04-24T14:34:22.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:34:22.516 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.516 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:22.516 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.516 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:22.516 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:22.516 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:22.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.516 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:22.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:34:22.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:34:22.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:34:22.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:34:22.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:34:22.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:34:22.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:34:22.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:34:22.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:34:22.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:22.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:34:22.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:34:22.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:22.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:34:22.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:34:22.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:22.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:34:22.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:34:22.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:22.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:34:22.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:34:22.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:34:22.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:34:22.521 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.521 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.521 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.521 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:22.521 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:22.521 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 119.867642585, 2024-04-24T14:34:22.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:34:22.521 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.521 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:22.522 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.522 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:22.522 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:22.522 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:34:22.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.522 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:22.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:22.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:34:22.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:22.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:34:22.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:34:22.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:34:22.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:34:22.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:34:22.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:34:22.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:34:22.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:34:22.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:34:22.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:34:22.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:22.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:34:22.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:34:22.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:22.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:34:22.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:34:22.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:22.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:34:22.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:34:22.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:22.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:34:22.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:34:22.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:34:22.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:34:22.527 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.527 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.527 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.527 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:22.527 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:22.527 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 119.867561063, 2024-04-24T14:34:22.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:34:22.527 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.527 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:22.527 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.527 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:22.528 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:22.528 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:34:22.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.528 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:22.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:22.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:34:22.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:22.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:34:22.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:34:22.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:34:22.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:34:22.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:34:22.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:34:22.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:34:22.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:34:22.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:34:22.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:34:22.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:22.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:34:22.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:34:22.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:22.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:34:22.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:34:22.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.532 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.533 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:22.533 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:22.533 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 119.435648716, 2024-04-24T14:34:22.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:34:22.533 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.533 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:22.533 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.533 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:22.533 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:22.533 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:34:22.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.534 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:22.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:22.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:34:22.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:34:22.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:34:22.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:34:22.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:34:22.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:22.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:34:22.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:34:22.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:22.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:34:22.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:34:22.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:22.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:34:22.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:34:22.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:22.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:34:22.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:34:22.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:34:22.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:34:22.538 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.538 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.538 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.539 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:22.539 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:22.539 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 119.43562041600001, 2024-04-24T14:34:22.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:34:22.539 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.539 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:22.539 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.539 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:22.539 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:22.539 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:34:22.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.539 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:34:22.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:34:22.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:34:22.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:34:22.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:34:22.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:34:22.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:34:22.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:34:22.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:34:22.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:34:22.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:22.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:34:22.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:34:22.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:22.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:34:22.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:34:22.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:22.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:34:22.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:34:22.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:22.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:34:22.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 119.435532712, 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:34:22.544 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.545 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:22.545 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.545 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:22.545 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:22.545 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:34:22.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.545 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:22.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:22.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:34:22.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:34:22.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:34:22.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:34:22.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:34:22.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:34:22.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:34:22.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:34:22.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:34:22.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:34:22.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:34:22.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:22.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:34:22.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:34:22.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:22.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:34:22.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:34:22.550 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.550 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.550 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.550 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:22.550 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:22.550 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 119.43540991099999, 2024-04-24T14:34:22.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:34:22.550 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.550 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:22.550 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:34:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:34:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:34:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:34:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:34:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:34:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:34:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:34:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:34:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:34:22.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:34:22.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:22.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:34:22.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:34:22.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:22.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:34:22.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:34:22.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:22.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:34:22.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:34:22.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:22.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:34:22.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:34:22.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:34:22.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:34:22.555 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.555 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.556 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.556 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:34:22.556 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:22.556 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.129165350999997, 2024-04-24T14:34:22.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:34:22.556 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.556 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:22.556 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.556 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:34:22.556 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:34:22.556 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:34:22.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.557 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:22.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:22.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:34:22.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:22.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:34:22.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:34:22.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:34:22.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:34:22.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:34:22.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:34:22.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:34:22.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:34:22.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:34:22.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:34:22.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:34:22.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:34:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:34:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:34:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:34:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:34:22.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:34:22.561 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.561 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.561 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.561 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:22.561 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:22.561 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 42.435041351999999, 2024-04-24T14:34:22.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:34:22.561 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.561 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:22.561 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.561 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:34:22.561 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:34:22.562 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:34:22.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.562 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:22.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:22.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:34:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:34:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:34:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:34:22.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:34:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:34:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:34:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:34:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:34:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:34:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:34:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:34:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:34:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:34:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:34:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:34:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:34:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:34:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:34:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:34:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:34:22.566 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.475681386000002, 2024-04-24T14:34:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:34:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:22.567 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:34:22.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:34:22.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:34:22.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:34:22.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:34:22.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:34:22.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:34:22.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:34:22.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:34:22.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:34:22.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:34:22.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:34:22.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:34:22.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:34:22.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:34:22.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:34:22.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:34:22.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:34:22.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:34:22.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:34:22.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:34:22.572 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.572 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.572 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.573 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:22.573 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:22.573 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.475600831000001, 2024-04-24T14:34:22.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:34:22.573 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.573 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:22.573 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.573 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:22.573 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:22.573 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:34:22.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.574 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:22.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:22.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:34:22.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:34:22.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:34:22.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:34:22.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:34:22.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:34:22.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:34:22.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:34:22.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:34:22.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:34:22.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:34:22.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:34:22.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:34:22.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:34:22.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:34:22.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:34:22.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:34:22.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:34:22.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:34:22.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:34:22.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:34:22.578 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.578 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.578 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.579 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:22.579 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:22.579 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.475553101999999, 2024-04-24T14:34:22.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:34:22.579 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.579 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:22.579 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.579 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:22.579 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:22.579 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:34:22.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.580 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:22.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:22.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:34:22.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:34:22.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:34:22.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:34:22.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:34:22.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:34:22.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:34:22.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:34:22.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:34:22.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:34:22.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:34:22.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:34:22.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:34:22.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:34:22.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:34:22.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:34:22.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:34:22.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:34:22.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:34:22.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:34:22.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:34:22.584 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.584 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.585 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.585 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:22.585 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:22.585 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.475525339000001, 2024-04-24T14:34:22.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:34:22.585 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.585 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:22.585 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.585 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:22.585 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:22.585 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:34:22.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.586 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:22.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:22.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:34:22.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:34:22.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:34:22.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:34:22.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:34:22.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:34:22.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:34:22.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:34:22.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:34:22.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:34:22.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:34:22.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:34:22.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:34:22.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:34:22.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:34:22.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:34:22.590 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.590 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.590 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.591 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:22.591 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:22.591 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.475485903999999, 2024-04-24T14:34:22.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:34:22.591 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.591 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:22.591 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.591 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:22.591 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:22.591 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:34:22.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.591 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:22.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:22.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:34:22.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:22.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:34:22.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:34:22.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:34:22.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:34:22.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:34:22.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:34:22.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:34:22.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:34:22.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:34:22.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:34:22.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:34:22.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:34:22.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:34:22.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:34:22.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:34:22.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:34:22.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:34:22.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:34:22.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:34:22.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:34:22.596 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.596 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.596 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.597 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:22.597 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:22.597 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.47546092, 2024-04-24T14:34:22.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:34:22.597 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.597 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:22.597 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.597 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:22.597 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:22.597 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:34:22.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.597 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:34:22.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:34:22.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:34:22.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:34:22.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:34:22.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:34:22.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:34:22.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:34:22.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:34:22.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:34:22.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:34:22.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:34:22.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:34:22.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:34:22.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:34:22.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:34:22.602 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.602 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.602 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.603 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:22.603 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:22.603 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 23.475435261000001, 2024-04-24T14:34:22.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:34:22.603 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:22.603 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:22.603 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:22.603 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:22.603 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:22.603 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:34:22.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.603 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:22.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:22.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:22.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:22.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:22.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:22.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:22.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:34:22.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:22.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:22.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:34:22.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:34:22.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:22.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:34:22.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:34:22.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:22.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:34:22.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:22.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:34:22.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:34:22.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:22.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:34:22.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:34:22.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:34:22.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:34:22.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:22.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:34:22.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:34:22.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:22.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:34:22.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:34:22.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:22.608 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:22.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:34:22.610 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:22.615+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:22.611 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:22.711 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:22.713+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:22.756 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:22.758 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:22.758 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:22.758 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:22.758 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:22.767 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:22.783 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:22.801 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:22.811 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:22.817+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:22.827 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:22.846 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:22.868 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:22.870 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:22.870 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:22.870 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:22.870 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:22.870 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:22.870 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:22.871 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:22.871 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:22.881 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:22.912 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:22.915+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:23.013 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:23.015+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:23.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T14:34:23.027 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:23.027 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:23.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:marked out osd.5. 2024-04-24T14:34:23.114 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:23.119+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:23.125 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:23.125 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:23.134 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:23.215 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:23.221+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:23.238 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:23.238 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:23.247 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:23.315 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:23.321+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:23.392 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:23.393 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:23.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:23.416 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:23.419+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:23.503 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:23.504 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:23.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:23.517 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:23.521+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:23.617 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:23.621+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:23.653 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:23.653 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:23.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:23.718 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:23.721+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:23.764 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:23.764 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:23.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:23.819 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:23.825+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:23.875 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:23.875 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:23.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:23.920 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:23.925+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:23.983 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:23.984 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:23.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:24.020 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:24.025+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:24.049 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-24T14:34:24.114 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:24.114 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:24.121 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:24.125+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:24.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:24.222 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:24.227+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:24.226 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:24.226 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:24.236 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:24.323 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:24.327+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:24.386 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:24.386 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:24.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:24.424 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:24.427+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:24.490 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:24.490 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:24.499 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:24.542 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:24.547+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:24.606 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:24.607 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:24.616 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:24.625 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:24.631+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:24.714 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:24.715 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:24.723 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:24.726 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:24.731+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:24.815 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:24.816 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:24.825 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:24.826 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:24.831+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:24.886 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:34:24.927 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:24.927 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:24.928 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:24.933+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:24.936 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:25.027 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:25.028 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:25.033+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:25.031 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:25.032 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:25.041 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:25.049 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:25.069 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:25.090 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:25.129 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:25.131+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:25.136 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:25.136 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:25.145 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:25.231 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:25.235+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:25.240 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:25.240 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:25.249 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:25.321 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:25.331 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:25.335+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:25.342 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:25.345 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:25.345 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:25.345 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:25.355 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:25.355 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:25.355 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:25.355 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:25.356 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:25.365 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:25.432 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:25.437+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:25.464 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:25.465 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:25.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:25.465 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:25.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.465 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:34:25.465 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:25.465 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.398036021000003, 2024-04-24T14:34:25.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:34:25.465 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.465 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:25.465 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:25.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:25.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:34:25.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:25.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:34:25.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:34:25.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:34:25.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:34:25.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:34:25.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:34:25.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:34:25.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:34:25.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:34:25.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:34:25.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:25.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:34:25.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:34:25.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:34:25.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:34:25.469 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.469 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.469 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.470 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:34:25.470 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:25.470 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.403221442, 2024-04-24T14:34:25.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:34:25.470 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.470 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:25.470 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:25.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:25.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:34:25.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:34:25.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:34:25.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:34:25.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:34:25.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:34:25.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:34:25.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:25.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:34:25.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:34:25.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:34:25.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:34:25.474 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.474 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.474 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.474 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:34:25.474 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:25.475 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.392292183999999, 2024-04-24T14:34:25.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:34:25.475 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.475 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:25.475 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:25.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:25.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:34:25.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:25.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:34:25.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:34:25.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:34:25.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:34:25.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:34:25.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:34:25.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:34:25.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:34:25.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:34:25.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:34:25.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:25.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:34:25.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:34:25.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:34:25.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:34:25.479 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.479 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.479 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.479 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:34:25.479 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:25.479 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.381336102999999, 2024-04-24T14:34:25.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:34:25.479 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.480 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:25.480 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:25.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:25.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:34:25.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:25.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:34:25.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:34:25.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:34:25.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:34:25.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:34:25.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:34:25.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:34:25.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:34:25.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:34:25.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:34:25.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:25.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:34:25.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:34:25.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:34:25.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:34:25.483 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.483 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.484 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.484 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:34:25.484 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:25.484 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.369766737999999, 2024-04-24T14:34:25.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:34:25.484 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.484 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:25.484 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:25.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:25.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:34:25.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:25.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:34:25.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:34:25.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:34:25.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:34:25.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:34:25.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:34:25.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:34:25.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:34:25.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:34:25.488 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.488 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.488 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.488 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:34:25.488 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:25.489 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.116647079, 2024-04-24T14:34:25.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:34:25.489 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.489 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:25.489 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:25.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:25.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:34:25.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:25.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:34:25.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:34:25.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:34:25.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:34:25.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:34:25.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:34:25.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:34:25.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:34:25.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:34:25.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:34:25.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:25.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:34:25.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:34:25.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.105901911, 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:25.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:25.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:34:25.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:25.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:34:25.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:34:25.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:34:25.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:34:25.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:34:25.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:34:25.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:34:25.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:34:25.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:34:25.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:34:25.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:25.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:34:25.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:34:25.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:34:25.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:34:25.497 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.497 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.497 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.497 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:34:25.497 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:25.498 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.093535634999999, 2024-04-24T14:34:25.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:34:25.498 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.498 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:25.498 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:25.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:25.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:34:25.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:25.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:34:25.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:34:25.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:34:25.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:34:25.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:34:25.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:34:25.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:34:25.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:34:25.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:34:25.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:34:25.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:25.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:34:25.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:34:25.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:34:25.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:34:25.502 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.502 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.502 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.502 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:34:25.502 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:25.502 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 30.968599514000001, 2024-04-24T14:34:25.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:34:25.502 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.503 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:25.503 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:25.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:25.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:34:25.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:25.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:34:25.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:34:25.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:34:25.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:34:25.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:34:25.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:34:25.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:34:25.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:34:25.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:34:25.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:34:25.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:25.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:34:25.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:34:25.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:34:25.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:34:25.506 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.506 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.506 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.507 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:34:25.507 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:25.507 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 30.949211671, 2024-04-24T14:34:25.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:34:25.507 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.507 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:25.507 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:25.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:34:25.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:34:25.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:34:25.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:34:25.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:34:25.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:34:25.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:34:25.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:34:25.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:25.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:34:25.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:34:25.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:34:25.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:34:25.511 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.511 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.511 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.511 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:34:25.511 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:25.511 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 30.939483828, 2024-04-24T14:34:25.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:34:25.512 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.512 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:25.512 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:25.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:25.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:34:25.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:25.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:34:25.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:34:25.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:34:25.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:34:25.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:34:25.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:34:25.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:34:25.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:34:25.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:34:25.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:34:25.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:25.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:34:25.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:34:25.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:34:25.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:34:25.515 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.515 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.515 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 30.779374147999999, 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:25.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:34:25.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:25.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:34:25.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:34:25.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:34:25.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:34:25.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:34:25.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:25.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:34:25.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:34:25.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:34:25.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:34:25.520 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.520 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.520 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.520 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:34:25.520 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:25.520 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 30.769645151999999, 2024-04-24T14:34:25.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:34:25.521 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.521 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:25.521 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:25.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:25.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:34:25.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:25.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:34:25.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:34:25.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:34:25.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:34:25.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:34:25.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:34:25.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:34:25.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:34:25.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:34:25.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:34:25.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:25.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:34:25.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:34:25.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:34:25.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:34:25.524 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.524 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.525 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.525 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:34:25.525 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:25.525 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.434674316999999, 2024-04-24T14:34:25.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:34:25.525 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.525 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:25.525 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:25.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:25.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:25.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:25.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:34:25.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:34:25.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:34:25.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:34:25.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:34:25.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:34:25.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:34:25.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:34:25.529 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.529 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.529 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.529 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:25.529 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:25.529 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.590523568999998, 2024-04-24T14:34:25.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:34:25.529 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.529 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:25.530 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:25.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:25.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:34:25.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:34:25.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:34:25.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:34:25.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:34:25.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:34:25.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:34:25.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:34:25.533 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.533 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.533 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.533 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:5 3.2 3.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:34:25.533 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:25.533 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.262285927000001, 2024-04-24T14:34:25.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043717790000000001, 2024-04-24T14:34:25.533 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.533 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-24T14:34:25.534 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:25.534 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:34:25.534 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:34:25.534 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 5 2024-04-24T14:34:25.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.534 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:25.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207678+0000", 2024-04-24T14:34:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999962 2024-04-24T14:34:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207700+0000", 2024-04-24T14:34:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1313000000000001e-05 2024-04-24T14:34:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207757+0000", 2024-04-24T14:34:25.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7499e-05 2024-04-24T14:34:25.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207779+0000", 2024-04-24T14:34:25.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1478999999999999e-05 2024-04-24T14:34:25.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.212054+0000", 2024-04-24T14:34:25.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042750849999999996 2024-04-24T14:34:25.536 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.537 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.537 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.537 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:25.537 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:25.537 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 7.3667605690000002, 2024-04-24T14:34:25.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044084859999999997, 2024-04-24T14:34:25.537 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.537 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:25.537 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103206+0000", 2024-04-24T14:34:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104731+0000", 2024-04-24T14:34:25.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001524755 2024-04-24T14:34:25.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104759+0000", 2024-04-24T14:34:25.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7973000000000001e-05 2024-04-24T14:34:25.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104807+0000", 2024-04-24T14:34:25.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8112000000000003e-05 2024-04-24T14:34:25.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104907+0000", 2024-04-24T14:34:25.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100317 2024-04-24T14:34:25.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104918+0000", 2024-04-24T14:34:25.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747999999999999e-05 2024-04-24T14:34:25.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.107616+0000", 2024-04-24T14:34:25.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026977559999999999 2024-04-24T14:34:25.541 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.541 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.541 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.541 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(3.2 83/82 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:25.541 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:25.541 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 7.3603661919999999, 2024-04-24T14:34:25.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041425760000000002, 2024-04-24T14:34:25.541 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.541 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:25.541 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:25.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:25.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109600+0000", 2024-04-24T14:34:25.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:25.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111210+0000", 2024-04-24T14:34:25.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016097279999999999 2024-04-24T14:34:25.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111237+0000", 2024-04-24T14:34:25.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7189999999999999e-05 2024-04-24T14:34:25.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111251+0000", 2024-04-24T14:34:25.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.365e-05 2024-04-24T14:34:25.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111367+0000", 2024-04-24T14:34:25.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000115703 2024-04-24T14:34:25.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111378+0000", 2024-04-24T14:34:25.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1873e-05 2024-04-24T14:34:25.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.113744+0000", 2024-04-24T14:34:25.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023657069999999999 2024-04-24T14:34:25.545 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.545 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.545 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.545 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:25.545 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:25.545 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 7.3434656159999996, 2024-04-24T14:34:25.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046193859999999996, 2024-04-24T14:34:25.545 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.546 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:25.546 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:25.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:25.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126501+0000", 2024-04-24T14:34:25.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:25.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128169+0000", 2024-04-24T14:34:25.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016686229999999999 2024-04-24T14:34:25.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128202+0000", 2024-04-24T14:34:25.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2539000000000001e-05 2024-04-24T14:34:25.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128222+0000", 2024-04-24T14:34:25.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0149e-05 2024-04-24T14:34:25.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128375+0000", 2024-04-24T14:34:25.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000153459 2024-04-24T14:34:25.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128399+0000", 2024-04-24T14:34:25.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3393000000000001e-05 2024-04-24T14:34:25.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.131122+0000", 2024-04-24T14:34:25.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027227140000000002 2024-04-24T14:34:25.549 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.549 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.549 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.549 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 49'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@49'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:25.549 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:25.549 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 7.3216842360000003, 2024-04-24T14:34:25.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038414529999999999, 2024-04-24T14:34:25.549 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:25.550 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:25.550 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:25.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:25.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:25.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:25.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:25.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:25.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:25.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148282+0000", 2024-04-24T14:34:25.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:25.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:25.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149875+0000", 2024-04-24T14:34:25.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015929900000000001 2024-04-24T14:34:25.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:25.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149908+0000", 2024-04-24T14:34:25.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3102000000000003e-05 2024-04-24T14:34:25.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149941+0000", 2024-04-24T14:34:25.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2480000000000001e-05 2024-04-24T14:34:25.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:25.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150031+0000", 2024-04-24T14:34:25.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0394000000000002e-05 2024-04-24T14:34:25.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:25.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150043+0000", 2024-04-24T14:34:25.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1967000000000001e-05 2024-04-24T14:34:25.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:25.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:25.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:25.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.152125+0000", 2024-04-24T14:34:25.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020817029999999999 2024-04-24T14:34:25.553 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.553 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.554 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.554 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:25.554 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:25.554 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:25.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:25.556 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:25.559+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:25.656 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:25.661+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:25.666 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:25.666 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:25.666 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:25.666 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:25.676 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:25.756 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:25.761+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:25.780 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:25.780 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:25.780 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:25.780 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:25.780 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:25.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:25.857 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:25.861+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:25.870 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:34:25.870 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^18955\b' 2024-04-24T14:34:25.870 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^18955\b' 2024-04-24T14:34:25.896 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:25.896 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:25.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:25.896 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.020225618000001, 2024-04-24T14:34:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:34:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:34:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:34:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:34:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:34:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:34:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:34:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:34:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:34:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:34:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:34:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:34:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:34:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:34:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:34:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:34:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:34:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:34:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:34:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:34:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:34:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:34:25.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:34:25.903 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.903 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.903 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.903 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:25.903 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:25.903 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.020196633000001, 2024-04-24T14:34:25.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:34:25.903 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:25.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:25.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:25.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:34:25.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:34:25.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:34:25.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:34:25.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:34:25.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:34:25.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:34:25.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:34:25.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:25.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:34:25.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:34:25.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:34:25.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:34:25.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:25.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:34:25.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:34:25.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.020178147999999, 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:25.909 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:34:25.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.910 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:25.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:25.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:34:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:34:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:34:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:34:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:34:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:34:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:34:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:34:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:34:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:25.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:34:25.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:34:25.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:34:25.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:34:25.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:25.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:34:25.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:34:25.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:34:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:34:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:34:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:34:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:34:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:34:25.914 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.915 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.915 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:25.915 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:25.915 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.020162867, 2024-04-24T14:34:25.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:34:25.915 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.915 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.915 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:25.915 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:25.915 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:25.915 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:34:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:25.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:25.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:34:25.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:34:25.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:34:25.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:34:25.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:34:25.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:34:25.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:34:25.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:34:25.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:25.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:34:25.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:34:25.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:34:25.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:34:25.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:34:25.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:34:25.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:34:25.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.920 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.921 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.921 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:25.921 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:25.921 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.020147086000001, 2024-04-24T14:34:25.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:34:25.921 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.921 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:25.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:25.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:25.921 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:34:25.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.922 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:25.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:25.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:34:25.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:34:25.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:25.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:34:25.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:34:25.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:34:25.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:34:25.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:34:25.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:34:25.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:34:25.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:34:25.926 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.926 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.926 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.927 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:25.927 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:25.927 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.020132797999999, 2024-04-24T14:34:25.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:34:25.927 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.927 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:25.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:25.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:25.927 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:34:25.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.927 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:34:25.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:34:25.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:34:25.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:34:25.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:34:25.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:34:25.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:34:25.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:34:25.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:25.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:34:25.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:34:25.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:34:25.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:34:25.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:25.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:34:25.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:34:25.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:34:25.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:34:25.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:34:25.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:34:25.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:34:25.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:34:25.932 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.932 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.932 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.932 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:25.932 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.020113045999999, 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:25.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:25.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:25.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:34:25.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:34:25.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:34:25.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:34:25.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:34:25.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:34:25.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:34:25.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:34:25.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:34:25.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:34:25.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:34:25.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:34:25.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:34:25.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:34:25.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:34:25.938 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.938 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.938 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.938 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:25.938 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:25.938 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.020095599000001, 2024-04-24T14:34:25.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:34:25.938 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.938 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:25.939 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:25.939 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:25.939 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:34:25.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.939 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:25.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:25.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:34:25.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:34:25.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:34:25.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:34:25.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:34:25.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:25.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:34:25.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:34:25.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:34:25.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:34:25.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:25.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:34:25.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:34:25.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.943 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.944 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.944 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4d9853ae:::rbd_id.INVALIDIMAGE:head, version: 53'12, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4d9853ae:::rbd_id.INVALIDIMAGE:head@53'12, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:25.944 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:25.944 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.994817068, 2024-04-24T14:34:25.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0075925020000000001, 2024-04-24T14:34:25.944 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.944 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.944 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:25.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905244+0000", 2024-04-24T14:34:25.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905243+0000", 2024-04-24T14:34:25.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:25.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905253+0000", 2024-04-24T14:34:25.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1619999999999997e-06 2024-04-24T14:34:25.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905262+0000", 2024-04-24T14:34:25.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2399999999999996e-06 2024-04-24T14:34:25.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.905268+0000", 2024-04-24T14:34:25.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2530000000000001e-06 2024-04-24T14:34:25.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912457+0000", 2024-04-24T14:34:25.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071890779999999998 2024-04-24T14:34:25.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912462+0000", 2024-04-24T14:34:25.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5159999999999996e-06 2024-04-24T14:34:25.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912836+0000", 2024-04-24T14:34:25.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037484699999999998 2024-04-24T14:34:25.948 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.948 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.948 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.948 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 62'66, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@62'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:25.948 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:25.948 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.971198138999998, 2024-04-24T14:34:25.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0036829689999999999, 2024-04-24T14:34:25.948 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.948 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.948 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928863+0000", 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.928862+0000", 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930264+0000", 2024-04-24T14:34:25.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014016059999999999 2024-04-24T14:34:25.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930281+0000", 2024-04-24T14:34:25.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7859999999999998e-05 2024-04-24T14:34:25.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930291+0000", 2024-04-24T14:34:25.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9299999999999998e-06 2024-04-24T14:34:25.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930894+0000", 2024-04-24T14:34:25.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060250899999999999 2024-04-24T14:34:25.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.930903+0000", 2024-04-24T14:34:25.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4599999999999992e-06 2024-04-24T14:34:25.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.932546+0000", 2024-04-24T14:34:25.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016425019999999999 2024-04-24T14:34:25.952 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.952 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.952 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.952 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head, version: 62'58, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head@62'58, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:25.952 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:25.952 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.965483191000001, 2024-04-24T14:34:25.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037783690000000002, 2024-04-24T14:34:25.952 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.952 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.952 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934578+0000", 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.934576+0000", 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:25.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936033+0000", 2024-04-24T14:34:25.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001456658 2024-04-24T14:34:25.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936071+0000", 2024-04-24T14:34:25.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7752e-05 2024-04-24T14:34:25.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936087+0000", 2024-04-24T14:34:25.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5919000000000001e-05 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936226+0000", 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013910800000000001 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.936240+0000", 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.456e-05 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938356+0000", 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021157540000000001 2024-04-24T14:34:25.955 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 62'66, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@62'66, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.964436663000001, 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0052678259999999998, 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:25.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935624+0000", 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.935623+0000", 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937055+0000", 2024-04-24T14:34:25.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014314480000000001 2024-04-24T14:34:25.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937085+0000", 2024-04-24T14:34:25.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0204000000000001e-05 2024-04-24T14:34:25.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.937097+0000", 2024-04-24T14:34:25.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1708e-05 2024-04-24T14:34:25.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938410+0000", 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013128910000000001 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.938419+0000", 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1740000000000006e-06 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.940892+0000", 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024734990000000001 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 62'82, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@62'82, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.954925272000001, 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003877447, 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945136+0000", 2024-04-24T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.945134+0000", 2024-04-24T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946323+0000", 2024-04-24T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001188635 2024-04-24T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946352+0000", 2024-04-24T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9238e-05 2024-04-24T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.946374+0000", 2024-04-24T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1396999999999999e-05 2024-04-24T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.947098+0000", 2024-04-24T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072458199999999998 2024-04-24T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.947109+0000", 2024-04-24T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0557000000000001e-05 2024-04-24T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.949013+0000", 2024-04-24T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019044929999999999 2024-04-24T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.948468287000001, 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041579369999999996, 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951593+0000", 2024-04-24T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.951585+0000", 2024-04-24T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999928 2024-04-24T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953339+0000", 2024-04-24T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017540019999999999 2024-04-24T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953363+0000", 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3178000000000001e-05 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953374+0000", 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1015e-05 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953522+0000", 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014857800000000001 2024-04-24T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953565+0000", 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2511e-05 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955751+0000", 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002185958 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 62'82, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@62'82, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.948053771000001, 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.005349749, 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952007+0000", 2024-04-24T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.952006+0000", 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953689+0000", 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00168326 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953717+0000", 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7994e-05 2024-04-24T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.953746+0000", 2024-04-24T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8059999999999999e-05 2024-04-24T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955798+0000", 2024-04-24T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002052089 2024-04-24T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.955812+0000", 2024-04-24T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2024-04-24T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.957357+0000", 2024-04-24T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001545537 2024-04-24T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.2 68/67 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.939263653000001, 2024-04-24T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0051567999999999996, 2024-04-24T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960797+0000", 2024-04-24T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.960796+0000", 2024-04-24T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962459+0000", 2024-04-24T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016622259999999999 2024-04-24T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962481+0000", 2024-04-24T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2359999999999999e-05 2024-04-24T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.962496+0000", 2024-04-24T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.523e-05 2024-04-24T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.964299+0000", 2024-04-24T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018025129999999999 2024-04-24T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.964316+0000", 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7436999999999999e-05 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.965954+0000", 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016381099999999999 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head, version: 62'116, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head@62'116, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.906257463999999, 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003994821, 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993804+0000", 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.993802+0000", 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994846+0000", 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00104368 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994884+0000", 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7796000000000002e-05 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994900+0000", 2024-04-24T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5917e-05 2024-04-24T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995046+0000", 2024-04-24T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014653499999999999 2024-04-24T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995055+0000", 2024-04-24T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0759999999999992e-06 2024-04-24T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.997798+0000", 2024-04-24T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0027432709999999998 2024-04-24T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(2.2 68/67 [PushOp(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 49'1, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@49'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.905950006000001, 2024-04-24T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0038339310000000001, 2024-04-24T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994111+0000", 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994110+0000", 2024-04-24T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994914+0000", 2024-04-24T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080383099999999997 2024-04-24T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994943+0000", 2024-04-24T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9475000000000002e-05 2024-04-24T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.994955+0000", 2024-04-24T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1525000000000001e-05 2024-04-24T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995123+0000", 2024-04-24T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016812899999999999 2024-04-24T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.995143+0000", 2024-04-24T14:34:25.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9724999999999998e-05 2024-04-24T14:34:25.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.997945+0000", 2024-04-24T14:34:25.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028023890000000002 2024-04-24T14:34:25.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.983 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.983 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head, version: 62'144, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head@62'144, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:25.983 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:25.984 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.865644810999999, 2024-04-24T14:34:25.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0041651270000000002, 2024-04-24T14:34:25.984 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.984 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.984 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:25.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034416+0000", 2024-04-24T14:34:25.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.034415+0000", 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035025+0000", 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060999700000000003 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035077+0000", 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2132999999999997e-05 2024-04-24T14:34:25.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035100+0000", 2024-04-24T14:34:25.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2690000000000001e-05 2024-04-24T14:34:25.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035293+0000", 2024-04-24T14:34:25.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019358900000000001 2024-04-24T14:34:25.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.035311+0000", 2024-04-24T14:34:25.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7952999999999999e-05 2024-04-24T14:34:25.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.038581+0000", 2024-04-24T14:34:25.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032700670000000002 2024-04-24T14:34:25.987 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.987 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.987 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.987 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.0 68/67 [PushOp(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head, version: 62'144, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head@62'144, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.851479376, 2024-04-24T14:34:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0039835970000000002, 2024-04-24T14:34:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048582+0000", 2024-04-24T14:34:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.048581+0000", 2024-04-24T14:34:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049792+0000", 2024-04-24T14:34:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012111579999999999 2024-04-24T14:34:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049823+0000", 2024-04-24T14:34:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1137999999999999e-05 2024-04-24T14:34:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.049838+0000", 2024-04-24T14:34:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5061e-05 2024-04-24T14:34:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.050005+0000", 2024-04-24T14:34:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016745400000000001 2024-04-24T14:34:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.050022+0000", 2024-04-24T14:34:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7022000000000001e-05 2024-04-24T14:34:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:03.052565+0000", 2024-04-24T14:34:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025428669999999999 2024-04-24T14:34:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:25.991 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:25.994 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:25.994 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:34:25.994 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *18955\b' 2024-04-24T14:34:25.994 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *18955\b' 2024-04-24T14:34:25.994 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:25.994 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:34:25.994 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:25.998+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:25.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:26.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T14:34:26.093 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:26.094+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:26.144 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:26.144 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:26.144 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:26.144 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:26.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:26.202 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:26.206+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:26.295 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:26.301+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:26.301 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:26.301 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:26.301 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:26.301 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:26.301 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:26.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:26.396 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:26.398+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:26.408 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:26.408 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:26.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:26.408 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:26.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.408 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:97 3.1 3:990e5750:::rbd_header.12ff64619e2d:head [watch reconnect cookie 140276853203408 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:26.408 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:26.408 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.059668721999998, 2024-04-24T14:34:26.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012161632, 2024-04-24T14:34:26.408 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.408 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 97 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352831+0000", 2024-04-24T14:34:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352830+0000", 2024-04-24T14:34:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352834+0000", 2024-04-24T14:34:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2699999999999998e-06 2024-04-24T14:34:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352844+0000", 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0406e-05 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.352850+0000", 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6439999999999997e-06 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362391+0000", 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095408969999999996 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.362683+0000", 2024-04-24T14:34:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029238000000000002 2024-04-24T14:34:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.364992+0000", 2024-04-24T14:34:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023092020000000002 2024-04-24T14:34:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:225 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.907226610000002, 2024-04-24T14:34:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010959015000000001, 2024-04-24T14:34:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 225 2024-04-24T14:34:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505273+0000", 2024-04-24T14:34:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505275+0000", 2024-04-24T14:34:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0269999999999998e-06 2024-04-24T14:34:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505279+0000", 2024-04-24T14:34:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4279999999999998e-06 2024-04-24T14:34:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.505282+0000", 2024-04-24T14:34:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.982e-06 2024-04-24T14:34:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515209+0000", 2024-04-24T14:34:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099266419999999994 2024-04-24T14:34:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.515466+0000", 2024-04-24T14:34:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025719700000000002 2024-04-24T14:34:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.516232+0000", 2024-04-24T14:34:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076608899999999996 2024-04-24T14:34:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.656194868, 2024-04-24T14:34:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:34:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:34:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:34:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:34:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:34:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:34:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:34:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:34:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:34:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:34:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:34:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:34:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:34:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:34:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:34:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:34:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:34:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.656144034, 2024-04-24T14:34:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:34:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:34:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:34:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:34:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:34:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:34:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:34:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:34:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:34:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:34:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:34:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:34:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:34:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:34:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:34:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:34:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.656107726999998, 2024-04-24T14:34:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:34:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:34:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:34:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:34:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:34:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:34:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:34:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:34:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:34:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:34:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:34:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:34:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:34:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:34:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:34:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:34:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:34:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.652696476999999, 2024-04-24T14:34:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:34:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:34:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:34:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:34:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:34:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:34:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:34:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:34:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:34:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:34:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:34:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:34:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:34:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:34:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:34:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.652634800000001, 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:34:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:34:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.652600223999997, 2024-04-24T14:34:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:34:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:34:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:34:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:34:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:34:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:34:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:34:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:34:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.652570885000003, 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:34:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:34:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:34:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:34:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:34:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:34:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:34:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:34:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:34:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:34:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:34:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:34:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:34:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:34:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:34:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:34:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:34:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:34:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.652494302999997, 2024-04-24T14:34:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:34:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:34:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:34:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:34:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:34:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:34:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:34:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:34:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:34:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:34:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:34:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:34:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:34:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:34:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:34:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:34:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:34:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:34:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.463 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.463 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:26.463 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.652446757999996, 2024-04-24T14:34:26.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:34:26.463 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.463 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.463 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.463 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.463 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.463 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:34:26.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.463 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:26.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:26.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:34:26.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:34:26.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:34:26.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:34:26.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:34:26.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:34:26.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:34:26.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:34:26.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:34:26.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:34:26.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:34:26.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:34:26.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:34:26.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:34:26.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:34:26.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:34:26.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:34:26.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.468 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.652406116000002, 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:26.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:26.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:26.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:34:26.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:34:26.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:34:26.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:34:26.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:34:26.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:34:26.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:34:26.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:34:26.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:34:26.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:34:26.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:34:26.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:34:26.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:34:26.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:34:26.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:34:26.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:34:26.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:34:26.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:34:26.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:34:26.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:34:26.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.474 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.475 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.475 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:26.475 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.652389005000003, 2024-04-24T14:34:26.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:34:26.475 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.475 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.475 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:34:26.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.475 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:26.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:26.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:34:26.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:26.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:34:26.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:34:26.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:34:26.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:34:26.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:34:26.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:34:26.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:34:26.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:34:26.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:34:26.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:34:26.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:34:26.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:34:26.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:34:26.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:34:26.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:34:26.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:34:26.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:34:26.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:34:26.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:34:26.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:34:26.480 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.480 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.480 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.481 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:481 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.481 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:26.481 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.618517298, 2024-04-24T14:34:26.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010661718000000001, 2024-04-24T14:34:26.481 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.481 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.481 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 481 2024-04-24T14:34:26.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.481 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:26.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793982+0000", 2024-04-24T14:34:26.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793981+0000", 2024-04-24T14:34:26.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793985+0000", 2024-04-24T14:34:26.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8469999999999998e-06 2024-04-24T14:34:26.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:26.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0220000000000004e-06 2024-04-24T14:34:26.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793998+0000", 2024-04-24T14:34:26.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8559999999999998e-06 2024-04-24T14:34:26.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803807+0000", 2024-04-24T14:34:26.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098088629999999993 2024-04-24T14:34:26.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.803916+0000", 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010854899999999999 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.804644+0000", 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072835399999999996 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:497 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.600576986, 2024-04-24T14:34:26.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013996471, 2024-04-24T14:34:26.486 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.486 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.486 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.486 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.486 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.486 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 497 2024-04-24T14:34:26.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.486 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:26.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811923+0000", 2024-04-24T14:34:26.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811922+0000", 2024-04-24T14:34:26.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:26.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811925+0000", 2024-04-24T14:34:26.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4120000000000001e-06 2024-04-24T14:34:26.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811936+0000", 2024-04-24T14:34:26.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1077000000000001e-05 2024-04-24T14:34:26.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.811941+0000", 2024-04-24T14:34:26.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7539999999999997e-06 2024-04-24T14:34:26.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822289+0000", 2024-04-24T14:34:26.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010347635000000001 2024-04-24T14:34:26.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.822457+0000", 2024-04-24T14:34:26.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000168257 2024-04-24T14:34:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.825919+0000", 2024-04-24T14:34:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003461979 2024-04-24T14:34:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:25 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.445019045000002, 2024-04-24T14:34:26.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013349789000000001, 2024-04-24T14:34:26.491 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.491 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:26.491 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.491 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:26.491 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:26.491 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 25 2024-04-24T14:34:26.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.491 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:26.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967480+0000", 2024-04-24T14:34:26.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967479+0000", 2024-04-24T14:34:26.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967669+0000", 2024-04-24T14:34:26.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000189829 2024-04-24T14:34:26.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967687+0000", 2024-04-24T14:34:26.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8199999999999999e-05 2024-04-24T14:34:26.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967695+0000", 2024-04-24T14:34:26.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4869999999999996e-06 2024-04-24T14:34:26.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.972501+0000", 2024-04-24T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048057509999999996 2024-04-24T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.972824+0000", 2024-04-24T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032322299999999998 2024-04-24T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.973071+0000", 2024-04-24T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024751299999999999 2024-04-24T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977624+0000", 2024-04-24T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0045527570000000002 2024-04-24T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980627+0000", 2024-04-24T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003002972 2024-04-24T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980743+0000", 2024-04-24T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2024-04-24T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980830+0000", 2024-04-24T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7534999999999995e-05 2024-04-24T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:26 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.444795310000003, 2024-04-24T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013403597999999999, 2024-04-24T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 26 2024-04-24T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:26.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967704+0000", 2024-04-24T14:34:26.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967703+0000", 2024-04-24T14:34:26.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967874+0000", 2024-04-24T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017088499999999999 2024-04-24T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967886+0000", 2024-04-24T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1744e-05 2024-04-24T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967893+0000", 2024-04-24T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2024-04-24T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974028+0000", 2024-04-24T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061356650000000002 2024-04-24T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974347+0000", 2024-04-24T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031897400000000003 2024-04-24T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:26.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.974774+0000", 2024-04-24T14:34:26.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042661700000000001 2024-04-24T14:34:26.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:26.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980930+0000", 2024-04-24T14:34:26.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061563360000000001 2024-04-24T14:34:26.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:26.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.980945+0000", 2024-04-24T14:34:26.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4864e-05 2024-04-24T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981035+0000", 2024-04-24T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9390999999999998e-05 2024-04-24T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981108+0000", 2024-04-24T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3255000000000003e-05 2024-04-24T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:27 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.444598163999999, 2024-04-24T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013394262000000001, 2024-04-24T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 27 2024-04-24T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:26.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967901+0000", 2024-04-24T14:34:26.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.967900+0000", 2024-04-24T14:34:26.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968061+0000", 2024-04-24T14:34:26.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016101900000000001 2024-04-24T14:34:26.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968072+0000", 2024-04-24T14:34:26.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1012999999999999e-05 2024-04-24T14:34:26.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968079+0000", 2024-04-24T14:34:26.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.46e-06 2024-04-24T14:34:26.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.976820+0000", 2024-04-24T14:34:26.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087406289999999998 2024-04-24T14:34:26.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.976951+0000", 2024-04-24T14:34:26.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131187 2024-04-24T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977129+0000", 2024-04-24T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000178135 2024-04-24T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981126+0000", 2024-04-24T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00399709 2024-04-24T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981193+0000", 2024-04-24T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7083999999999997e-05 2024-04-24T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981251+0000", 2024-04-24T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7754000000000003e-05 2024-04-24T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981296+0000", 2024-04-24T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4765000000000003e-05 2024-04-24T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:28 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.444412931999999, 2024-04-24T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013427247999999999, 2024-04-24T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 28 2024-04-24T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968087+0000", 2024-04-24T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968086+0000", 2024-04-24T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968247+0000", 2024-04-24T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016121199999999999 2024-04-24T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968260+0000", 2024-04-24T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.275e-05 2024-04-24T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968266+0000", 2024-04-24T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3770000000000001e-06 2024-04-24T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977699+0000", 2024-04-24T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094331439999999992 2024-04-24T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.977894+0000", 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194945 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.978258+0000", 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036359400000000002 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981140+0000", 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028821540000000001 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981360+0000", 2024-04-24T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022056300000000001 2024-04-24T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981444+0000", 2024-04-24T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3733999999999995e-05 2024-04-24T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.981514+0000", 2024-04-24T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9702000000000005e-05 2024-04-24T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4937.0:29 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.444226086999997, 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014492283, 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4937", 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3456578899", 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 29 2024-04-24T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968273+0000", 2024-04-24T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968434+0000", 2024-04-24T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016130199999999999 2024-04-24T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968484+0000", 2024-04-24T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0003000000000002e-05 2024-04-24T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.968499+0000", 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4628e-05 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979299+0000", 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010800624999999999 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979488+0000", 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000189037 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.979784+0000", 2024-04-24T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029562000000000002 2024-04-24T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982305+0000", 2024-04-24T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025212339999999998 2024-04-24T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982680+0000", 2024-04-24T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000374762 2024-04-24T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982731+0000", 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1146999999999997e-05 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:53.982766+0000", 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4755000000000002e-05 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:26.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:26.523 INFO:tasks.workunit.client.0.smithi173.stderr:+ LOG_FILE=/tmp/tmp.waDYLkcsIV/rbd-nbd.log 2024-04-24T14:34:26.523 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -n '' ']' 2024-04-24T14:34:26.524 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map --quiesce rbd/testrbdnbd16809 --log-file=/tmp/tmp.waDYLkcsIV/rbd-nbd.log 2024-04-24T14:34:26.524 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:34:26.524 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:34:26.524 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:34:26.524 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:34:26.524 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:34:26.524 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:34:26.524 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce rbd/testrbdnbd16809 --log-file=/tmp/tmp.waDYLkcsIV/rbd-nbd.log 2024-04-24T14:34:26.525 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:26.525+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:26.602 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd0 2024-04-24T14:34:26.602 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:34:26.602 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:26.602 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:26.603 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:26.603 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:26.622 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:26.626+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:26.632 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:26.632 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:26.632 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:26.632 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:26.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:26.667 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=19140 2024-04-24T14:34:26.667 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 19140 2024-04-24T14:34:26.667 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 19140 -C rbd-nbd 2024-04-24T14:34:26.673 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:26.673 INFO:tasks.workunit.client.0.smithi173.stdout: 19140 ? 00:00:00 rbd-nbd 2024-04-24T14:34:26.673 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo mkfs /dev/nbd0 2024-04-24T14:34:26.674 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:26.674 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:26.675 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:26.676 INFO:tasks.workunit.client.0.smithi173.stderr:++ which mkfs 2024-04-24T14:34:26.678 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/sbin/mkfs 2024-04-24T14:34:26.678 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:26.678 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/sbin/mkfs /dev/nbd0 2024-04-24T14:34:26.723 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:26.726+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:26.727 INFO:tasks.workunit.client.0.smithi173.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-04-24T14:34:26.738 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:26.739 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:26.739 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:26.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:26.824 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:26.826+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:26.901 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:192 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:26.901 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.433820283000003, 2024-04-24T14:34:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010036329, 2024-04-24T14:34:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 192 2024-04-24T14:34:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:26.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471100+0000", 2024-04-24T14:34:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471099+0000", 2024-04-24T14:34:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471103+0000", 2024-04-24T14:34:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8800000000000001e-06 2024-04-24T14:34:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471111+0000", 2024-04-24T14:34:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2930000000000001e-06 2024-04-24T14:34:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471116+0000", 2024-04-24T14:34:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1800000000000004e-06 2024-04-24T14:34:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480338+0000", 2024-04-24T14:34:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092213069999999998 2024-04-24T14:34:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.480467+0000", 2024-04-24T14:34:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012901500000000001 2024-04-24T14:34:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481136+0000", 2024-04-24T14:34:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066948400000000003 2024-04-24T14:34:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.433797247000001, 2024-04-24T14:34:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:34:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:34:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:34:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:34:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:34:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:34:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:34:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:34:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:34:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:34:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:34:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:34:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:34:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:34:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:34:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.058440544, 2024-04-24T14:34:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:34:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:34:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:34:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:34:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:34:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:34:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:34:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:34:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:34:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:34:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:34:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:34:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:34:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:34:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:526 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3670016~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.057053021000002, 2024-04-24T14:34:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010568783999999999, 2024-04-24T14:34:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 526 2024-04-24T14:34:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847867+0000", 2024-04-24T14:34:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847866+0000", 2024-04-24T14:34:26.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847871+0000", 2024-04-24T14:34:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1590000000000001e-06 2024-04-24T14:34:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847883+0000", 2024-04-24T14:34:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.172e-05 2024-04-24T14:34:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847890+0000", 2024-04-24T14:34:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6950000000000003e-06 2024-04-24T14:34:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857418+0000", 2024-04-24T14:34:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095285909999999995 2024-04-24T14:34:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857588+0000", 2024-04-24T14:34:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169541 2024-04-24T14:34:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858436+0000", 2024-04-24T14:34:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084814700000000005 2024-04-24T14:34:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.057012389999997, 2024-04-24T14:34:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:34:26.922 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:34:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:34:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:34:26.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:34:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:34:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:34:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:34:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:34:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:34:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:34:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:34:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:34:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:34:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:34:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.926 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.056985300999997, 2024-04-24T14:34:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:34:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:34:26.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:34:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:34:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:34:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:34:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:34:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:34:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:34:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:34:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:34:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:34:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:34:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:34:26.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:34:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:26.932 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.056961334, 2024-04-24T14:34:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:34:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:34:26.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:34:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:34:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:34:26.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:34:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:34:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:34:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:34:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:34:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:34:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:34:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:34:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:34:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:34:26.937 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.042977354999998, 2024-04-24T14:34:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:34:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:34:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:34:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:34:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:34:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:34:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:34:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:34:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:34:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:34:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:34:26.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:34:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:34:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:34:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:34:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:34:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:34:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:34:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:34:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:34:26.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:34:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:34:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:34:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:26.944 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.042861043999999, 2024-04-24T14:34:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:34:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:34:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:34:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:34:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:34:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:34:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:34:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:34:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:34:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:34:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:34:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:34:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:34:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:34:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:34:26.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:34:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:34:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:34:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:34:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:34:26.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:34:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:34:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:34:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:26.950 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.042813385999999, 2024-04-24T14:34:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:34:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:34:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:34:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:34:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:34:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:34:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:34:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:34:26.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:34:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:34:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:34:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:34:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:34:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:34:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:34:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:34:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:34:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:34:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:34:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:34:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:34:26.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:34:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:34:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:26.958 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.038776087999999, 2024-04-24T14:34:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:34:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:34:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:26.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:34:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:34:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:34:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:34:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:34:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:34:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:34:26.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:34:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:34:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:34:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:34:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:34:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:34:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:34:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:34:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:34:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:34:26.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:34:26.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:34:26.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:34:26.964 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.964 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.964 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.965 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.965 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:26.965 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.038712328999999, 2024-04-24T14:34:26.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:34:26.965 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.965 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.965 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.965 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.965 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.965 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:34:26.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.965 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:26.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:26.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:34:26.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:34:26.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:26.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:34:26.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:34:26.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:34:26.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:34:26.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:34:26.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:34:26.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:34:26.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:34:26.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:34:26.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:34:26.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:34:26.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:34:26.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:34:26.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:34:26.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:34:26.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:34:26.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:34:26.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:34:26.970 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.971 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.971 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.971 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.971 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:26.971 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.038686839, 2024-04-24T14:34:26.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:34:26.971 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.971 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.972 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:34:26.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.972 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:26.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:26.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:26.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:26.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:26.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:34:26.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:34:26.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:34:26.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:34:26.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:34:26.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:34:26.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:34:26.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:34:26.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:34:26.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:34:26.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:34:26.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:34:26.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:34:26.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:34:26.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:34:26.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:34:26.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:34:26.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:34:26.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:34:26.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.977 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.977 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.977 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:26.977 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.038663089000003, 2024-04-24T14:34:26.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:34:26.978 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.978 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.978 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:34:26.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.978 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:26.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:26.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:34:26.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:34:26.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:34:26.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:34:26.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:34:26.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:34:26.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:34:26.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:34:26.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:34:26.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:34:26.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:34:26.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:34:26.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:34:26.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:34:26.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:34:26.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:34:26.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:34:26.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:34:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:34:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:34:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:34:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.984 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.984 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:26.984 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.038624575999997, 2024-04-24T14:34:26.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:34:26.984 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.984 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.984 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:34:26.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.984 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.038596378999998, 2024-04-24T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:26.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:26.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:34:26.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:26.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:34:26.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:34:26.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:34:26.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:34:26.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:34:26.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:34:26.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:34:26.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:34:26.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:26.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:34:26.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:34:26.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:34:26.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:34:26.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:26.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:34:26.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:34:26.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:34:26.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:34:26.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:26.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:34:26.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:34:26.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:26.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:34:26.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:34:26.995 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.996 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:26.996 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:26.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.996 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:26.996 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:26.996 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.038574922000002, 2024-04-24T14:34:26.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:34:26.996 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:26.996 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:26.996 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:26.996 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:26.996 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:26.997 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:34:26.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.997 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:26.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:26.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:26.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:26.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:26.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:26.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:26.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:34:26.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:26.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:26.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:34:26.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:34:26.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:26.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:34:26.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:34:26.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:26.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:34:26.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:26.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:26.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:26.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:26.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:34:26.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:34:27.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:27.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:34:27.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:34:27.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:27.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:34:27.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:34:27.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:27.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:34:27.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:34:27.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:27.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:34:27.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:34:27.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:27.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:34:27.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:34:27.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:27.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:34:27.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:34:27.002 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:27.002 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:27.002 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:27.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.002 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:27.002 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:27.002 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.038556079999999, 2024-04-24T14:34:27.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:34:27.003 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:27.003 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:27.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:27.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:27.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:27.003 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:34:27.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.003 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:27.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:27.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:27.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:34:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:561 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.99659346, 2024-04-24T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01066806, 2024-04-24T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 561 2024-04-24T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908327+0000", 2024-04-24T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908326+0000", 2024-04-24T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908329+0000", 2024-04-24T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.802e-06 2024-04-24T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908338+0000", 2024-04-24T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6260000000000001e-06 2024-04-24T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.908345+0000", 2024-04-24T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1389999999999998e-06 2024-04-24T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.917978+0000", 2024-04-24T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0096329840000000007 2024-04-24T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918178+0000", 2024-04-24T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019999199999999999 2024-04-24T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.918995+0000", 2024-04-24T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081708099999999999 2024-04-24T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.825723930999999, 2024-04-24T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:27.021 INFO:tasks.workunit.client.0.smithi173.stdout:Discarding device blocks: 4096/16384 done 2024-04-24T14:34:27.021 INFO:tasks.workunit.client.0.smithi173.stdout:Creating filesystem with 16384 4k blocks and 16384 inodes 2024-04-24T14:34:27.021 INFO:tasks.workunit.client.0.smithi173.stdout: 2024-04-24T14:34:27.021 INFO:tasks.workunit.client.0.smithi173.stdout:Allocating group tables: 0/1 done 2024-04-24T14:34:27.022 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:27.022 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:27.022+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:27.029 INFO:tasks.workunit.client.0.smithi173.stdout:Writing inode tables: 0/1 done 2024-04-24T14:34:27.065 INFO:tasks.workunit.client.0.smithi173.stdout:Writing superblocks and filesystem accounting information: 0/1 done 2024-04-24T14:34:27.065 INFO:tasks.workunit.client.0.smithi173.stdout: 2024-04-24T14:34:27.067 INFO:tasks.workunit.client.0.smithi173.stderr:+ mkdir /tmp/tmp.waDYLkcsIV/mnt 2024-04-24T14:34:27.069 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo mount /dev/nbd0 /tmp/tmp.waDYLkcsIV/mnt 2024-04-24T14:34:27.070 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:27.070 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:27.070 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-24T14:34:27.072 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:27.072 INFO:tasks.workunit.client.0.smithi173.stderr:++ which mount 2024-04-24T14:34:27.076 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/mount 2024-04-24T14:34:27.076 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:27.076 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/mount /dev/nbd0 /tmp/tmp.waDYLkcsIV/mnt 2024-04-24T14:34:27.121 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:27.122+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:27.125 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd snap create rbd/testrbdnbd16809@quiesce3 2024-04-24T14:34:27.127 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:27.128 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:27.128 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:27.128 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:27.137 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:27.222 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:27.225+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:27.234 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:27.234 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:27.234 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:27.234 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:27.234 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:27.243 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:27.323 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:27.329+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:27.359 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:27.359 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:27.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:27.359 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:27.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.359 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:27.359 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:27.359 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 124.73663416700001, 2024-04-24T14:34:27.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:34:27.359 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.359 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:27.359 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.359 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:27.360 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:27.360 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:34:27.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.360 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:27.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:27.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:34:27.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:27.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:34:27.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:34:27.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:34:27.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:34:27.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:34:27.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:34:27.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:34:27.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:34:27.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.364 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.365 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:27.365 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:27.365 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 124.73645707999999, 2024-04-24T14:34:27.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:34:27.365 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.365 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:27.365 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.365 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:27.365 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:27.365 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:34:27.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.365 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:34:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:27.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:34:27.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:34:27.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:34:27.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:34:27.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:27.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:34:27.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:34:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:34:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:34:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 124.736353166, 2024-04-24T14:34:27.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:34:27.371 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.371 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:27.371 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.371 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:27.371 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:27.371 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:34:27.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.371 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:27.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:34:27.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:34:27.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:27.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:34:27.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:34:27.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:27.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:34:27.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:34:27.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:34:27.375 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.376 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.376 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.376 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:27.376 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:27.376 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 124.73596471499999, 2024-04-24T14:34:27.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:34:27.376 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.376 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:27.376 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.376 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:27.376 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:27.376 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:34:27.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:27.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:34:27.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:34:27.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:34:27.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:34:27.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:34:27.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:34:27.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:34:27.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:34:27.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:34:27.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:34:27.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:27.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:34:27.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:34:27.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:27.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:34:27.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:34:27.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:27.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:34:27.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:34:27.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:27.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:34:27.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:34:27.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:34:27.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:34:27.381 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.381 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.382 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.382 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:27.382 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:27.382 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 124.73587950300001, 2024-04-24T14:34:27.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:34:27.382 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.382 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:27.382 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.382 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:27.382 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:27.382 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:27.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.383 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:27.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:27.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:34:27.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:27.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:34:27.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:34:27.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:34:27.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:34:27.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:34:27.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:34:27.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:34:27.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:34:27.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:34:27.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:34:27.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:27.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:34:27.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:34:27.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:27.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:34:27.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:34:27.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:27.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:34:27.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:34:27.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:27.389 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:27.390 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 124.735642622, 2024-04-24T14:34:27.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:34:27.390 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.390 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:27.390 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.390 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:27.390 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:27.390 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:34:27.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.390 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:27.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:27.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:34:27.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:27.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:34:27.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:34:27.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:34:27.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:34:27.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:34:27.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:34:27.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:34:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:34:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:34:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:34:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:34:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:34:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:27.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:34:27.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:34:27.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:27.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:34:27.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:34:27.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:27.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:34:27.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:34:27.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:34:27.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:34:27.395 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.395 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.395 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.395 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:27.395 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:27.395 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 124.7355611, 2024-04-24T14:34:27.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:34:27.396 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.396 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:27.396 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.396 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:27.396 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:27.396 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:34:27.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.396 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:27.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:34:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:34:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:34:27.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:34:27.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:34:27.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:34:27.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:34:27.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:27.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:34:27.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:34:27.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:27.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:34:27.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:34:27.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:27.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:34:27.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:34:27.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:34:27.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:34:27.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.401 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.401 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:27.401 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:27.401 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 124.303648753, 2024-04-24T14:34:27.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:34:27.401 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.402 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:27.402 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.402 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:27.402 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:27.402 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:34:27.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.402 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:27.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:27.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:34:27.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:27.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:34:27.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:34:27.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:34:27.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:34:27.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:34:27.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:34:27.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:34:27.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:34:27.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:34:27.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:34:27.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:27.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:34:27.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:34:27.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:27.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:34:27.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:34:27.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:27.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:34:27.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:34:27.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:27.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:34:27.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:34:27.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:34:27.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:34:27.409 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.409 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.409 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.409 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:27.409 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:27.409 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 124.30362045299999, 2024-04-24T14:34:27.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:34:27.409 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.409 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:27.409 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.409 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:27.409 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:27.410 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:34:27.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.410 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:27.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:27.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:34:27.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:27.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:34:27.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:34:27.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:34:27.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:34:27.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:27.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:34:27.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:34:27.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:27.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:34:27.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:34:27.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:27.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:34:27.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:34:27.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:27.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:34:27.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:34:27.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:34:27.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:34:27.414 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.414 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.415 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.415 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:27.415 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:27.415 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 124.303532749, 2024-04-24T14:34:27.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:34:27.415 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.415 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:27.415 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.415 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:27.415 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:27.416 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:34:27.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.416 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:27.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:27.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:34:27.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:27.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:34:27.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:34:27.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:34:27.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:34:27.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:34:27.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:34:27.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:34:27.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:34:27.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:34:27.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:34:27.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:27.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:34:27.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:34:27.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:27.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:34:27.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:34:27.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:34:27.420 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.421 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.421 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.421 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:27.421 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:27.421 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 124.303409948, 2024-04-24T14:34:27.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:34:27.421 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.421 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:27.421 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.421 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:27.421 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:27.421 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:34:27.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.422 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:27.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:27.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:34:27.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:27.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:34:27.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:34:27.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:34:27.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:34:27.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:34:27.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:34:27.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:34:27.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:34:27.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:34:27.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:34:27.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:27.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:34:27.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:34:27.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:27.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:34:27.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:34:27.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:27.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:34:27.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:34:27.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:27.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:34:27.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:34:27.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:34:27.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:34:27.428 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.428 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.428 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.428 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:34:27.428 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:27.428 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.997165387999999, 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:27.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:34:27.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:27.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:34:27.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:34:27.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:34:27.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:34:27.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:34:27.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:34:27.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:34:27.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:34:27.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:27.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:34:27.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:34:27.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:34:27.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:34:27.433 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.433 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.433 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.433 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:27.433 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:27.433 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 47.303041389000001, 2024-04-24T14:34:27.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:34:27.434 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.434 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:27.434 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.434 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:34:27.434 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:34:27.434 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:34:27.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.434 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:27.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:27.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:34:27.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:27.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:34:27.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:34:27.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:34:27.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:34:27.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:34:27.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:34:27.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:34:27.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:34:27.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:27.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:34:27.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:34:27.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:34:27.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:34:27.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:27.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:34:27.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:34:27.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:34:27.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:34:27.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.343681423, 2024-04-24T14:34:27.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:34:27.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:34:27.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:34:27.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:34:27.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:34:27.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:34:27.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:27.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:34:27.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:34:27.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:34:27.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:34:27.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:27.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:34:27.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:34:27.445 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.445 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.445 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.445 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:27.445 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:27.445 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.343600867999999, 2024-04-24T14:34:27.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:34:27.445 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.445 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:27.446 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.446 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:27.446 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:27.446 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:34:27.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.446 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:27.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:27.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:34:27.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:27.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:34:27.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:34:27.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:34:27.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:34:27.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:34:27.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:34:27.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:34:27.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:34:27.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:27.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:34:27.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:34:27.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:34:27.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:34:27.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:27.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:34:27.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:34:27.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:34:27.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:34:27.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:34:27.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:34:27.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:34:27.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:34:27.452 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.452 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.452 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.453 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:27.453 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:27.453 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.343553139000001, 2024-04-24T14:34:27.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:34:27.453 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.453 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:27.453 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.453 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:27.453 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:27.453 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:34:27.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:34:27.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:27.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:34:27.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:34:27.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:34:27.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:34:27.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:34:27.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:34:27.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:34:27.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:34:27.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:27.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:34:27.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:34:27.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:34:27.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:34:27.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:27.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:34:27.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:34:27.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:34:27.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:34:27.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:34:27.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:34:27.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:34:27.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:34:27.459 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.459 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.459 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.343525375999999, 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:34:27.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:34:27.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:34:27.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:34:27.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:34:27.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:34:27.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:34:27.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:34:27.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:27.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:34:27.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:34:27.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:34:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:34:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:34:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:34:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:34:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.465 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:27.466 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:27.466 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.343485941000001, 2024-04-24T14:34:27.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:34:27.466 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.466 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:27.466 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.466 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:27.466 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:27.466 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:34:27.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.466 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:27.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:27.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:34:27.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:27.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:27.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:34:27.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:34:27.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:34:27.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:34:27.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:34:27.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:34:27.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.343460957000001, 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.471 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:27.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:34:27.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:34:27.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:34:27.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:34:27.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:34:27.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:34:27.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:27.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:34:27.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:34:27.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:34:27.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:34:27.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:27.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:34:27.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:34:27.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.476 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 28.343435297999999, 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:27.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:27.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:27.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:27.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:27.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:27.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:27.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:34:27.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:27.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:34:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:34:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:34:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:34:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:34:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:34:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:34:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:34:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:34:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:27.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:34:27.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:34:27.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:27.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:27.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:27.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:34:27.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:34:27.482 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.482 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.482 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.482 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:27.482 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:27.482 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:27.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:34:27.485 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:27.485 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:27.489+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:27.585 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:27.586+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:27.589 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:27.589 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:27.589 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:27.589 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:27.598 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:27.627 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:27.648 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:27.669 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:27.686 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:27.690+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:27.692 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:27.697 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:27.698 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:27.698 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:27.707 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:27.715 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:27.736 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:27.738 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:27.738 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:27.738 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:27.738 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:27.738 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:27.738 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:27.798 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:27.802+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:27.806 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:27.806 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:27.815 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:27.888 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:27.893+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:27.916 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:27.916 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:27.925 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:27.988 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:27.990+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:28.024 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:28.024 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:28.041 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:28.056 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 1] 2024-04-24T14:34:28.056 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:34:28.056 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-24T14:34:28.056 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.284354 2024-04-24T14:34:28.056 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 0.28435405509710965 2024-04-24T14:34:28.089 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:28.093+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:28.093 INFO:tasks.workunit.client.0.smithi173.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-04-24T14:34:28.102 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo dd if=/tmp/tmp.waDYLkcsIV/data of=/tmp/tmp.waDYLkcsIV/mnt/test bs=1M count=1 oflag=direct 2024-04-24T14:34:28.103 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:28.103 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:28.105 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:28.105 INFO:tasks.workunit.client.0.smithi173.stderr:++ which dd 2024-04-24T14:34:28.107 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/dd 2024-04-24T14:34:28.108 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:28.108 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.waDYLkcsIV/data of=/tmp/tmp.waDYLkcsIV/mnt/test bs=1M count=1 oflag=direct 2024-04-24T14:34:28.122 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records in 2024-04-24T14:34:28.122 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records out 2024-04-24T14:34:28.122 INFO:tasks.workunit.client.0.smithi173.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00387957 s, 270 MB/s 2024-04-24T14:34:28.123 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo umount /tmp/tmp.waDYLkcsIV/mnt 2024-04-24T14:34:28.123 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:28.123 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:28.126 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:28.126 INFO:tasks.workunit.client.0.smithi173.stderr:++ which umount 2024-04-24T14:34:28.129 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/umount 2024-04-24T14:34:28.129 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:28.129 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.waDYLkcsIV/mnt 2024-04-24T14:34:28.137 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:28.137 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:28.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:28.160 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device /dev/nbd0 19140 2024-04-24T14:34:28.160 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=/dev/nbd0 2024-04-24T14:34:28.160 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=19140 2024-04-24T14:34:28.160 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:34:28.160 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:28.160 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:28.162 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:28.162 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:28.165 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:28.165 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:28.165 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:34:28.189 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:28.190+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:28.245 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:28.245 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:28.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:28.261 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:34:28.262 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^19140\b' 2024-04-24T14:34:28.262 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^19140\b' 2024-04-24T14:34:28.290 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:28.293+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:28.318 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:28.319 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *19140\b' 2024-04-24T14:34:28.319 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:34:28.319 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *19140\b' 2024-04-24T14:34:28.331 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:28.331 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:34:28.361 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:28.361 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:28.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:28.391 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:28.394+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:28.469 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:28.469 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:28.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:28.492 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:28.497+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:28.593 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:28.597+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:28.629 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:28.629 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:28.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:28.694 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:28.698+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:28.738 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:28.739 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:28.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:28.795 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:28.801+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:28.833 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:34:28.833 INFO:tasks.workunit.client.0.smithi173.stderr:+ cat /tmp/tmp.waDYLkcsIV/rbd-nbd.log 2024-04-24T14:34:28.834 INFO:tasks.workunit.client.0.smithi173.stdout:2024-04-24T14:34:26.537+0000 7f780e257140 0 ceph version 17.2.7-915-g330067db (330067dbce47655d1f762ef14d546d2fde16a6a9) quincy (stable), process rbd-nbd, pid 19131 2024-04-24T14:34:28.834 INFO:tasks.workunit.client.0.smithi173.stdout:2024-04-24T14:34:26.537+0000 7f780e257140 0 pidfile_write: ignore empty --pid-file 2024-04-24T14:34:28.834 INFO:tasks.workunit.client.0.smithi173.stdout:2024-04-24T14:34:26.641+0000 7f780c58d700 0 -- 172.21.15.173:0/3159276725 >> [v2:172.21.15.90:6800/34413,v1:172.21.15.90:6801/34413] conn(0x562b4cc4cef0 msgr2=0x562b4cc4f2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T14:34:28.834 INFO:tasks.workunit.client.0.smithi173.stdout:2024-04-24T14:34:26.753+0000 7f780c58d700 0 -- 172.21.15.173:0/3159276725 >> [v2:172.21.15.90:6800/34413,v1:172.21.15.90:6801/34413] conn(0x7f77f8038390 msgr2=0x7f77f803a770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T14:34:28.835 INFO:tasks.workunit.client.0.smithi173.stdout:2024-04-24T14:34:28.245+0000 7f77f0ff9700 0 rbd-nbd: disconnect request received 2024-04-24T14:34:28.835 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep 'quiesce failed' /tmp/tmp.waDYLkcsIV/rbd-nbd.log 2024-04-24T14:34:28.835 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep 'quiesce failed' /tmp/tmp.waDYLkcsIV/rbd-nbd.log 2024-04-24T14:34:28.837 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:28.838 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd --options try-netlink,show-cookie map rbd/testrbdnbd16809 2024-04-24T14:34:28.838 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:34:28.838 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:34:28.840 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:34:28.840 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:34:28.843 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:34:28.843 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:34:28.843 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options try-netlink,show-cookie map rbd/testrbdnbd16809 2024-04-24T14:34:28.845 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:28.845 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:28.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:28.896 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:28.898+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:28.957 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:28.957 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:28.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:29.001 INFO:tasks.workunit.client.0.smithi173.stderr:+ OUT=/dev/nbd0 2024-04-24T14:34:29.001 INFO:tasks.workunit.client.0.smithi173.stderr:+ read DEV COOKIE 2024-04-24T14:34:29.002 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:34:29.002 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:29.002 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:29.002 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:29.003 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:29.010 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:29.014+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:29.077 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=19260 2024-04-24T14:34:29.077 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 19260 2024-04-24T14:34:29.077 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 19260 -C rbd-nbd 2024-04-24T14:34:29.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T14:34:29.086 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:29.086 INFO:tasks.workunit.client.0.smithi173.stdout: 19260 ? 00:00:00 rbd-nbd 2024-04-24T14:34:29.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo mount /dev/nbd0 /tmp/tmp.waDYLkcsIV/mnt 2024-04-24T14:34:29.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:29.087 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:29.089 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:29.089 INFO:tasks.workunit.client.0.smithi173.stderr:++ which mount 2024-04-24T14:34:29.090 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.1 primary-affinity to 0.284354 (8186352) 2024-04-24T14:34:29.092 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/mount 2024-04-24T14:34:29.092 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:29.092 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/mount /dev/nbd0 /tmp/tmp.waDYLkcsIV/mnt 2024-04-24T14:34:29.097 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:29.098+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:29.117 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:29.117 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:29.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:29.197 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:29.201+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:29.264 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:29.264 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:29.274 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:29.319 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:29.325+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:29.384 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:29.384 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:29.394 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:29.399 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:29.402+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:29.497 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:29.497 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:29.500 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:29.502+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:29.506 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:29.601 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:29.605+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:29.606 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:29.606 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:29.615 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:29.702 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:29.706+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:29.714 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:29.715 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:29.724 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:29.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:34:29.802 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:29.805+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:29.823 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:29.823 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:29.832 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:29.895 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:29.903 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:29.909+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:29.917 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:29.937 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:29.960 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:29.983 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:29.987 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:29.988 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:29.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:29.988 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:29.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.988 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:29.988 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:29.988 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.326958152000003, 2024-04-24T14:34:29.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:34:29.988 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:29.988 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:29.989 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:29.989 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:29.989 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:29.989 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:34:29.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.989 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:29.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:29.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:29.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:29.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:29.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:29.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:29.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:29.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:29.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:29.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:29.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:34:29.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:34:29.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:29.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:34:29.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:34:29.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:29.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:34:29.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:34:29.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:29.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:34:29.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:34:29.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:29.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:34:29.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:34:29.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:29.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:34:29.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:34:29.993 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:29.993 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:29.993 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:29.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.993 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:29.993 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:29.993 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.323923626000003, 2024-04-24T14:34:29.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:34:29.993 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:29.993 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:29.993 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:29.993 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:29.993 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:29.994 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:34:29.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.994 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:29.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:29.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:29.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:29.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:29.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:29.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:29.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:29.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:29.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:29.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:29.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:34:29.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:34:29.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:29.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:34:29.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:34:29.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:29.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:34:29.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:34:29.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:29.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:34:29.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:34:29.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:29.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:34:29.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:34:29.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:29.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:29.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:34:29.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:34:29.999 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:29.999 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:29.999 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:29.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:29.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.000 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:30.000 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:30.000 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.323899439999998, 2024-04-24T14:34:30.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:34:30.000 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.000 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:30.000 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:30.000 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:30.000 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:30.000 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:34:30.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.000 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:30.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:30.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:30.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:30.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:34:30.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:34:30.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:34:30.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:34:30.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:34:30.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:34:30.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:34:30.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:34:30.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:34:30.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:34:30.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:34:30.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:34:30.004 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.004 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.004 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.004 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:30.004 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:30.004 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.323882056000002, 2024-04-24T14:34:30.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:34:30.005 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.005 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:30.005 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:30.005 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:30.005 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:30.005 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:34:30.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.005 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:30.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:30.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:34:30.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:30.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:34:30.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:34:30.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:34:30.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:34:30.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:34:30.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:34:30.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.309445963999998, 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.009 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:30.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:30.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:34:30.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:30.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:34:30.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:34:30.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:34:30.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:34:30.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:34:30.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:34:30.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:34:30.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:34:30.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:34:30.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:34:30.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:34:30.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.309413665000001, 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:30.013 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:34:30.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.014 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:30.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:30.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:34:30.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:30.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:34:30.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:34:30.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:34:30.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:34:30.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:34:30.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:30.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:34:30.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:34:30.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:34:30.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:34:30.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:34:30.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:34:30.017 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.017 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.017 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.017 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:30.017 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:30.017 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.295161387999997, 2024-04-24T14:34:30.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:34:30.018 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.018 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:30.018 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:30.018 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:30.018 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:30.018 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:34:30.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.018 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:30.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:30.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:34:30.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:30.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:34:30.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:34:30.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:34:30.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:34:30.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:34:30.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:34:30.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:34:30.021 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.021 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.021 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.022 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:30.022 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:30.022 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.295132287999998, 2024-04-24T14:34:30.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:34:30.022 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.022 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:30.022 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:30.022 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:30.022 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:30.022 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:34:30.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:34:30.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:34:30.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:34:30.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:34:30.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:34:30.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:34:30.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:34:30.026 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.026 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.026 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.026 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:30.026 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:30.026 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.295106357999998, 2024-04-24T14:34:30.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:34:30.026 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:34:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:34:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:34:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:34:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:34:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:34:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:34:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:34:30.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:34:30.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:34:30.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:34:30.030 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.030 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.030 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.031 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:30.031 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:30.031 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.279385894000001, 2024-04-24T14:34:30.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:34:30.031 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.031 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:30.031 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:30.031 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:30.031 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:30.031 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:34:30.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.031 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:30.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:30.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:34:30.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:30.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:34:30.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:34:30.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.033 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:34:30.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:34:30.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.033 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:34:30.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:34:30.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:34:30.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:34:30.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:34:30.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:34:30.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:34:30.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:34:30.035 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.035 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.035 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.035 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:30.035 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:30.035 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.279353962000002, 2024-04-24T14:34:30.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:34:30.036 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.036 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:30.036 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:30.036 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:30.036 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:30.036 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:34:30.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.036 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:30.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:34:30.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:34:30.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:34:30.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:34:30.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:34:30.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:34:30.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:34:30.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:34:30.039 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.039 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.039 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.039 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:30.039 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:30.039 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.279286446, 2024-04-24T14:34:30.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:34:30.040 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.040 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:30.040 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:30.040 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:30.040 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:30.040 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:34:30.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.040 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:30.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:30.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:34:30.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:30.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:34:30.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:34:30.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.042 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:34:30.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:34:30.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.042 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:34:30.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:34:30.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.042 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:34:30.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:34:30.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:34:30.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:34:30.043 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.043 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.043 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.043 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:34:30.043 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:30.044 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 48.221873563000003, 2024-04-24T14:34:30.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:34:30.044 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.044 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.044 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:30.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:30.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:34:30.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:34:30.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:34:30.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:34:30.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:34:30.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:34:30.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:34:30.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:34:30.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:34:30.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:34:30.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:34:30.047 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.047 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.047 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.048 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:103 3.1 e87/74 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head v 87'261, mlcod=87'258)", 2024-04-24T14:34:30.048 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:30.048 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.0305923589999999, 2024-04-24T14:34:30.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019745728000000001, 2024-04-24T14:34:30.048 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.048 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.048 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:30.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960679+0000", 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960692+0000", 2024-04-24T14:34:30.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3709999999999999e-05 2024-04-24T14:34:30.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960722+0000", 2024-04-24T14:34:30.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9791999999999999e-05 2024-04-24T14:34:30.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960733+0000", 2024-04-24T14:34:30.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.128e-05 2024-04-24T14:34:30.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960857+0000", 2024-04-24T14:34:30.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000123981 2024-04-24T14:34:30.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960926+0000", 2024-04-24T14:34:30.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8881999999999995e-05 2024-04-24T14:34:30.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.051 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.051 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980353+0000", 2024-04-24T14:34:30.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019426853000000001 2024-04-24T14:34:30.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980426+0000", 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2803000000000001e-05 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:105 3.1 e87/74 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head v 87'263, mlcod=87'258)", 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.0214658249999999, 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014864743999999999, 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.052 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.053 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:30.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:30.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969805+0000", 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969820+0000", 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5116999999999999e-05 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969852+0000", 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2135999999999999e-05 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969864+0000", 2024-04-24T14:34:30.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.189e-05 2024-04-24T14:34:30.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972122+0000", 2024-04-24T14:34:30.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022580389999999999 2024-04-24T14:34:30.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972165+0000", 2024-04-24T14:34:30.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2429000000000003e-05 2024-04-24T14:34:30.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984619+0000", 2024-04-24T14:34:30.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012454735 2024-04-24T14:34:30.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984672+0000", 2024-04-24T14:34:30.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2077999999999998e-05 2024-04-24T14:34:30.056 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.056 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.056 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.056 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:112 3.1 e87/74 3:a983798d:::rbd_data.12ff64619e2d.0000000000000006:head v 87'265, mlcod=87'258)", 2024-04-24T14:34:30.057 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:30.057 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.0111269890000001, 2024-04-24T14:34:30.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012821978, 2024-04-24T14:34:30.057 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.057 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.057 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:30.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980144+0000", 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980159+0000", 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4769999999999999e-05 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980189+0000", 2024-04-24T14:34:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0419000000000001e-05 2024-04-24T14:34:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980213+0000", 2024-04-24T14:34:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4036000000000002e-05 2024-04-24T14:34:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980452+0000", 2024-04-24T14:34:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00023921500000000001 2024-04-24T14:34:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980468+0000", 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5843999999999999e-05 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992932+0000", 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012463314 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992968+0000", 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6075999999999998e-05 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.060 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.061 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:113 3.1 e87/74 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head v 87'267, mlcod=87'258)", 2024-04-24T14:34:30.061 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:30.061 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.007592265, 2024-04-24T14:34:30.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010561022999999999, 2024-04-24T14:34:30.061 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.061 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.061 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:30.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:30.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983679+0000", 2024-04-24T14:34:30.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:30.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983699+0000", 2024-04-24T14:34:30.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0392000000000001e-05 2024-04-24T14:34:30.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983737+0000", 2024-04-24T14:34:30.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8340000000000002e-05 2024-04-24T14:34:30.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983756+0000", 2024-04-24T14:34:30.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8402000000000001e-05 2024-04-24T14:34:30.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984706+0000", 2024-04-24T14:34:30.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095024400000000005 2024-04-24T14:34:30.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984741+0000", 2024-04-24T14:34:30.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4721000000000002e-05 2024-04-24T14:34:30.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994211+0000", 2024-04-24T14:34:30.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094708559999999997 2024-04-24T14:34:30.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994241+0000", 2024-04-24T14:34:30.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9941e-05 2024-04-24T14:34:30.065 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.065 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.065 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.065 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:115 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:30.065 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:30.065 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.0071382280000001, 2024-04-24T14:34:30.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010347242, 2024-04-24T14:34:30.066 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.066 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.066 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:30.066 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:34:30.066 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:30.066 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:30.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.066 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:30.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984132+0000", 2024-04-24T14:34:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984142+0000", 2024-04-24T14:34:30.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7370000000000001e-06 2024-04-24T14:34:30.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984154+0000", 2024-04-24T14:34:30.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2288e-05 2024-04-24T14:34:30.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984163+0000", 2024-04-24T14:34:30.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3749999999999992e-06 2024-04-24T14:34:30.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984246+0000", 2024-04-24T14:34:30.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2962999999999998e-05 2024-04-24T14:34:30.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984473+0000", 2024-04-24T14:34:30.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00022696399999999999 2024-04-24T14:34:30.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:30.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984771+0000", 2024-04-24T14:34:30.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00029739499999999999 2024-04-24T14:34:30.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:30.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992998+0000", 2024-04-24T14:34:30.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082275750000000009 2024-04-24T14:34:30.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:30.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994299+0000", 2024-04-24T14:34:30.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013005289999999999 2024-04-24T14:34:30.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994403+0000", 2024-04-24T14:34:30.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010376499999999999 2024-04-24T14:34:30.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994482+0000", 2024-04-24T14:34:30.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9013999999999999e-05 2024-04-24T14:34:30.071 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.071 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.071 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.071 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:128 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:30.071 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:30.072 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 2.9859472999999999, 2024-04-24T14:34:30.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011784144, 2024-04-24T14:34:30.072 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.072 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.072 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:30.072 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:34:30.072 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:30.072 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 128 2024-04-24T14:34:30.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.072 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:30.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:30.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005324+0000", 2024-04-24T14:34:30.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:30.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005332+0000", 2024-04-24T14:34:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2989999999999997e-06 2024-04-24T14:34:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005354+0000", 2024-04-24T14:34:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1705999999999999e-05 2024-04-24T14:34:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005362+0000", 2024-04-24T14:34:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5599999999999996e-06 2024-04-24T14:34:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005418+0000", 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6685999999999999e-05 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005535+0000", 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117008 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005732+0000", 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019718700000000001 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:30.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016440+0000", 2024-04-24T14:34:30.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010707787999999999 2024-04-24T14:34:30.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:30.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016913+0000", 2024-04-24T14:34:30.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047270400000000001 2024-04-24T14:34:30.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017047+0000", 2024-04-24T14:34:30.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013434699999999999 2024-04-24T14:34:30.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017109+0000", 2024-04-24T14:34:30.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2192000000000004e-05 2024-04-24T14:34:30.077 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.077 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.077 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.077 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:130 3.1 e87/74 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 87'269, mlcod=87'265)", 2024-04-24T14:34:30.078 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:30.078 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 2.9854945399999999, 2024-04-24T14:34:30.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013945631999999999, 2024-04-24T14:34:30.078 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.078 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.078 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:30.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:30.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005777+0000", 2024-04-24T14:34:30.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:30.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005791+0000", 2024-04-24T14:34:30.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3743999999999999e-05 2024-04-24T14:34:30.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005803+0000", 2024-04-24T14:34:30.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2330999999999999e-05 2024-04-24T14:34:30.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005810+0000", 2024-04-24T14:34:30.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8990000000000004e-06 2024-04-24T14:34:30.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005888+0000", 2024-04-24T14:34:30.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7872000000000003e-05 2024-04-24T14:34:30.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005911+0000", 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3156e-05 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019642+0000", 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013730456 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019724+0000", 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1981e-05 2024-04-24T14:34:30.081 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.082 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.082 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.082 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.082 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.082 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:30.083 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:30.084 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:30.085 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:30.089+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:30.085 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:30.085 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:30.085 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:30.085 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:30.085 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:30.085 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:30.094 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-24T14:34:30.108 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device detach rbd/testrbdnbd16809 --device-type nbd 2024-04-24T14:34:30.109 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:30.109 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:30.111 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:30.112 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:30.114 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:30.114 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:30.114 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd16809 --device-type nbd 2024-04-24T14:34:30.185 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:30.186+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:30.218 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false get_pid rbd 2024-04-24T14:34:30.218 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:34:30.218 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:30.218 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:30.219 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:30.221 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:30.236 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:30.236 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:30.236 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:30.236 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:30.246 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:30.285 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:30.289+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:30.289 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID= 2024-04-24T14:34:30.289 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n '' 2024-04-24T14:34:30.289 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 1 2024-04-24T14:34:30.289 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:30.289 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd16809 --device-type nbd 2024-04-24T14:34:30.289 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd16809 --device-type nbd 2024-04-24T14:34:30.289 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:30.290 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:30.291 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:30.291 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:30.293 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:30.293 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:30.294 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 rbd/testrbdnbd16809 --device-type nbd 2024-04-24T14:34:30.328 INFO:tasks.workunit.client.0.smithi173.stderr:rbd: could not validate attach request 2024-04-24T14:34:30.328 INFO:tasks.workunit.client.0.smithi173.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-04-24T14:34:30.328 INFO:tasks.workunit.client.0.smithi173.stderr:rbd: must specify --cookie or --force to proceed 2024-04-24T14:34:30.330 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:30.330 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -n '' ']' 2024-04-24T14:34:30.331 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd16809 --device-type nbd --force 2024-04-24T14:34:30.331 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:30.331 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:30.331 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:30.332 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:30.333 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:30.333 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:30.333 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 rbd/testrbdnbd16809 --device-type nbd --force 2024-04-24T14:34:30.350 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:30.350 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:30.350 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:30.350 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:30.350 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:30.359 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:30.403 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:30.409+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:30.462 INFO:tasks.workunit.client.0.smithi173.stdout:/dev/nbd0 2024-04-24T14:34:30.471 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:34:30.471 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:30.471 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:30.472 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:30.473 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:30.487 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:30.493+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:30.507 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:30.507 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:30.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:30.508 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:30.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.508 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:34:30.508 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:30.508 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.439911324999997, 2024-04-24T14:34:30.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:34:30.508 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.508 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.508 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:30.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:30.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:34:30.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:30.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:34:30.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:34:30.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:34:30.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:34:30.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:34:30.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:34:30.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:34:30.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:34:30.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:34:30.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:34:30.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:34:30.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:34:30.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:34:30.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:34:30.513 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.513 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.513 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.513 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:34:30.513 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:30.513 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.445096745999997, 2024-04-24T14:34:30.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:34:30.514 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.514 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.514 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:30.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:30.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:34:30.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:30.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:34:30.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:34:30.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:34:30.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:34:30.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:34:30.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:34:30.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:34:30.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:34:30.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:34:30.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:34:30.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:34:30.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:34:30.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:34:30.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:34:30.518 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.518 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.518 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.518 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:34:30.519 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:30.519 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.434167488, 2024-04-24T14:34:30.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:34:30.519 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.519 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.519 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:30.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:30.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:34:30.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:30.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:34:30.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:34:30.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:34:30.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:34:30.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:34:30.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:34:30.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:34:30.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:34:30.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:34:30.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:34:30.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:34:30.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:34:30.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:34:30.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:34:30.523 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.524 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.524 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.524 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:34:30.524 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:30.524 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.423211406999997, 2024-04-24T14:34:30.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:34:30.524 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.524 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.524 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:30.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:30.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:34:30.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:30.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:34:30.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:34:30.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:34:30.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:34:30.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:34:30.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:34:30.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:34:30.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:34:30.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:34:30.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:34:30.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:34:30.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:34:30.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:34:30.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:34:30.529 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.529 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.529 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.529 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:34:30.529 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:30.529 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.411642041999997, 2024-04-24T14:34:30.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:34:30.530 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.530 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.530 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:30.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:30.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:34:30.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:30.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:34:30.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:34:30.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:34:30.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:34:30.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:34:30.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:34:30.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:34:30.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:34:30.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:34:30.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:34:30.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:34:30.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:34:30.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:34:30.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:34:30.534 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.534 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.534 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.534 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:34:30.534 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:30.535 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.158522382999998, 2024-04-24T14:34:30.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:34:30.535 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.535 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.535 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:30.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:30.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:34:30.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:30.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:34:30.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:34:30.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:34:30.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:34:30.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:34:30.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:34:30.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:34:30.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:34:30.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:34:30.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:34:30.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:34:30.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:34:30.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:34:30.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:34:30.539 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.539 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.539 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.540 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:34:30.540 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:30.540 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.147777214999998, 2024-04-24T14:34:30.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:34:30.540 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.540 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.540 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:30.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:30.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:34:30.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:30.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:34:30.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:34:30.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:34:30.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:34:30.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:34:30.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:34:30.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:34:30.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:34:30.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:34:30.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:34:30.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:34:30.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:34:30.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:34:30.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:34:30.545 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.545 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.545 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.545 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:34:30.545 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:30.545 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.135410939000003, 2024-04-24T14:34:30.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:34:30.545 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.545 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.545 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:30.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:30.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:34:30.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:30.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:34:30.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:34:30.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:34:30.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:34:30.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:34:30.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:34:30.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:34:30.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:34:30.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:34:30.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:34:30.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:34:30.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:34:30.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:34:30.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:34:30.550 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.550 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.550 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.550 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:34:30.550 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:30.550 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.010474817999999, 2024-04-24T14:34:30.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:34:30.551 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.551 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.551 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:30.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:30.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:34:30.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:30.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:34:30.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:34:30.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:34:30.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:34:30.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:34:30.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:34:30.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:34:30.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:34:30.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:34:30.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:34:30.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:34:30.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:34:30.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:34:30.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:34:30.555 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.555 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.555 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.556 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:34:30.556 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:30.556 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 35.991086975000002, 2024-04-24T14:34:30.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:34:30.556 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.556 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.556 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:30.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:30.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:34:30.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:30.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:34:30.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:34:30.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:34:30.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:34:30.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:34:30.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:34:30.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:34:30.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:34:30.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:34:30.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:34:30.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:34:30.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:34:30.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:34:30.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:34:30.561 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.561 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.561 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.561 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:34:30.561 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:30.561 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 35.981359132000001, 2024-04-24T14:34:30.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:34:30.562 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.562 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.562 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:30.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:30.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:34:30.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:30.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:34:30.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:34:30.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:34:30.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:34:30.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:34:30.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:34:30.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:34:30.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:34:30.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:34:30.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:34:30.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:34:30.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:34:30.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:34:30.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:34:30.566 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.566 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.566 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.567 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:34:30.567 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:30.567 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 35.821249452000004, 2024-04-24T14:34:30.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:34:30.567 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.567 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.567 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:30.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:30.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:34:30.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:30.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:34:30.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:34:30.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:34:30.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:34:30.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:34:30.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:34:30.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:34:30.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:34:30.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:34:30.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:34:30.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:34:30.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:34:30.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:34:30.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:34:30.572 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.572 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.572 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.572 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:34:30.572 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:30.572 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 35.811520455999997, 2024-04-24T14:34:30.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:34:30.572 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.573 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.573 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:30.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:30.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:34:30.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:30.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:34:30.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:34:30.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:34:30.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:34:30.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:34:30.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:34:30.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:34:30.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:34:30.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:34:30.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:34:30.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:34:30.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:34:30.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:34:30.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:34:30.577 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.577 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.577 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.577 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:34:30.578 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:30.578 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.476549621, 2024-04-24T14:34:30.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:34:30.578 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.578 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:30.578 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:30.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:30.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:30.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:30.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:34:30.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:34:30.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:34:30.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:34:30.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:34:30.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:34:30.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:34:30.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:34:30.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:34:30.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:34:30.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:30.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:34:30.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:34:30.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:34:30.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:34:30.582 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.582 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.582 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.583 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:30.583 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:30.583 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.632398873, 2024-04-24T14:34:30.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:34:30.583 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.583 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:30.583 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:30.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:30.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:30.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:30.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:34:30.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:34:30.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:34:30.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:34:30.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:34:30.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:34:30.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:34:30.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:34:30.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:34:30.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:34:30.587 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.587 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.587 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.587 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:5 3.2 3.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:34:30.587 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:30.587 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.304161230999998, 2024-04-24T14:34:30.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043717790000000001, 2024-04-24T14:34:30.587 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.587 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-24T14:34:30.587 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:30.588 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:34:30.588 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:34:30.588 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 5 2024-04-24T14:34:30.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.588 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:30.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:30.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207678+0000", 2024-04-24T14:34:30.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999962 2024-04-24T14:34:30.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207700+0000", 2024-04-24T14:34:30.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1313000000000001e-05 2024-04-24T14:34:30.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207757+0000", 2024-04-24T14:34:30.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7499e-05 2024-04-24T14:34:30.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207779+0000", 2024-04-24T14:34:30.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1478999999999999e-05 2024-04-24T14:34:30.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.212054+0000", 2024-04-24T14:34:30.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042750849999999996 2024-04-24T14:34:30.591 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.591 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.591 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.591 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:30.592 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:30.592 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.408635873, 2024-04-24T14:34:30.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044084859999999997, 2024-04-24T14:34:30.592 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.592 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:30.592 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:30.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:30.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103206+0000", 2024-04-24T14:34:30.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:30.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104731+0000", 2024-04-24T14:34:30.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001524755 2024-04-24T14:34:30.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104759+0000", 2024-04-24T14:34:30.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7973000000000001e-05 2024-04-24T14:34:30.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104807+0000", 2024-04-24T14:34:30.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8112000000000003e-05 2024-04-24T14:34:30.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104907+0000", 2024-04-24T14:34:30.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100317 2024-04-24T14:34:30.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104918+0000", 2024-04-24T14:34:30.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747999999999999e-05 2024-04-24T14:34:30.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.107616+0000", 2024-04-24T14:34:30.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026977559999999999 2024-04-24T14:34:30.596 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.596 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.596 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.596 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(3.2 83/82 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:30.596 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:30.597 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.402241496, 2024-04-24T14:34:30.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041425760000000002, 2024-04-24T14:34:30.597 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.597 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:30.597 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:30.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:30.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109600+0000", 2024-04-24T14:34:30.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:30.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111210+0000", 2024-04-24T14:34:30.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016097279999999999 2024-04-24T14:34:30.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111237+0000", 2024-04-24T14:34:30.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7189999999999999e-05 2024-04-24T14:34:30.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111251+0000", 2024-04-24T14:34:30.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.365e-05 2024-04-24T14:34:30.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111367+0000", 2024-04-24T14:34:30.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000115703 2024-04-24T14:34:30.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111378+0000", 2024-04-24T14:34:30.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1873e-05 2024-04-24T14:34:30.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.113744+0000", 2024-04-24T14:34:30.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023657069999999999 2024-04-24T14:34:30.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.601 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.601 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:30.601 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:30.601 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.385340920000001, 2024-04-24T14:34:30.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046193859999999996, 2024-04-24T14:34:30.602 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.602 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:30.602 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:30.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126501+0000", 2024-04-24T14:34:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128169+0000", 2024-04-24T14:34:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016686229999999999 2024-04-24T14:34:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128202+0000", 2024-04-24T14:34:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2539000000000001e-05 2024-04-24T14:34:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128222+0000", 2024-04-24T14:34:30.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0149e-05 2024-04-24T14:34:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128375+0000", 2024-04-24T14:34:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000153459 2024-04-24T14:34:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128399+0000", 2024-04-24T14:34:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3393000000000001e-05 2024-04-24T14:34:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.131122+0000", 2024-04-24T14:34:30.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027227140000000002 2024-04-24T14:34:30.606 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.606 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.606 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.606 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 49'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@49'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:30.606 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:30.606 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.363559540000001, 2024-04-24T14:34:30.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038414529999999999, 2024-04-24T14:34:30.607 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:30.607 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:30.607 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:30.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:30.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:30.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:30.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:30.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:30.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:30.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148282+0000", 2024-04-24T14:34:30.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:30.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:30.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149875+0000", 2024-04-24T14:34:30.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015929900000000001 2024-04-24T14:34:30.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:30.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149908+0000", 2024-04-24T14:34:30.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3102000000000003e-05 2024-04-24T14:34:30.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149941+0000", 2024-04-24T14:34:30.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2480000000000001e-05 2024-04-24T14:34:30.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:30.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150031+0000", 2024-04-24T14:34:30.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0394000000000002e-05 2024-04-24T14:34:30.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:30.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150043+0000", 2024-04-24T14:34:30.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1967000000000001e-05 2024-04-24T14:34:30.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:30.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:30.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:30.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.152125+0000", 2024-04-24T14:34:30.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020817029999999999 2024-04-24T14:34:30.611 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.611 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.611 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.611 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:30.611 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:30.611 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:30.613 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=19331 2024-04-24T14:34:30.613 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 19331 2024-04-24T14:34:30.613 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 19331 -C rbd-nbd 2024-04-24T14:34:30.613 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:30.614 INFO:tasks.workunit.client.0.smithi173.stdout: 19331 ? 00:00:00 rbd-nbd 2024-04-24T14:34:30.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:30.615 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:30.617+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:30.616 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device detach /dev/nbd0 --device-type nbd 2024-04-24T14:34:30.616 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:30.616 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:30.616 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:30.616 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:30.616 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:30.616 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:30.616 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd0 --device-type nbd 2024-04-24T14:34:30.657 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false get_pid rbd 2024-04-24T14:34:30.657 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:34:30.657 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:30.657 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:30.658 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:30.658 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:30.714 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:30.717+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:30.727 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:30.727 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:30.727 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:30.731 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID= 2024-04-24T14:34:30.732 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n '' 2024-04-24T14:34:30.732 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 1 2024-04-24T14:34:30.732 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:30.732 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -n '' ']' 2024-04-24T14:34:30.732 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device attach --device /dev/nbd0 rbd/testrbdnbd16809 --device-type nbd --force 2024-04-24T14:34:30.732 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:30.732 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:30.733 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:30.734 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:30.735 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:30.735 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:30.735 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 rbd/testrbdnbd16809 --device-type nbd --force 2024-04-24T14:34:30.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:30.815 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:30.821+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:30.838 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:30.838 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:30.838 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:30.839 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:30.839 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:30.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:30.868 INFO:tasks.workunit.client.0.smithi173.stdout:/dev/nbd0 2024-04-24T14:34:30.878 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:34:30.878 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:30.878 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:30.879 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:30.879 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:30.916 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:30.921+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:30.949 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:30.949 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:30.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:30.949 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:30.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.949 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:30.949 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:30.949 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.072373763000002, 2024-04-24T14:34:30.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:34:30.950 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:30.950 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:30.950 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:30.950 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:30.950 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:30.950 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:34:30.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.950 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:30.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:30.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:30.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:30.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:30.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:30.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:34:30.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:30.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:30.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:34:30.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:34:30.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:30.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:34:30.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:34:30.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:34:30.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:34:30.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:34:30.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:34:30.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:30.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:34:30.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:34:30.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:34:30.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:34:30.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:30.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:34:30.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:34:30.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:34:30.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:34:30.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:30.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:34:30.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:34:30.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:30.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:34:30.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:34:30.956 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:30.956 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:30.956 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:30.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.956 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:30.956 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:30.956 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.072344778000001, 2024-04-24T14:34:30.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:34:30.957 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:30.957 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:30.957 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:30.957 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:30.957 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:30.957 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:34:30.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.957 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:30.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:30.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:30.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:30.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:30.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:30.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:30.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:30.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:30.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:34:30.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:34:30.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:30.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:34:30.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:34:30.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:34:30.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:34:30.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:34:30.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:34:30.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:30.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:34:30.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:34:30.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:34:30.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:34:30.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:30.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:34:30.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:34:30.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:34:30.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:34:30.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:30.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:34:30.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:34:30.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:30.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:34:30.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:34:30.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:30.963 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:30.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:30.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.963 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:30.963 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:30.963 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.072326293, 2024-04-24T14:34:30.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:34:30.963 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:30.964 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:30.964 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:30.964 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:30.964 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:30.964 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:34:30.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.964 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:30.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:30.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:30.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:30.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:30.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:30.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:34:30.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:30.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:30.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:34:30.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:34:30.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:30.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:34:30.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:34:30.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:34:30.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:34:30.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:34:30.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:34:30.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:30.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:34:30.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:34:30.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:34:30.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:34:30.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:30.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:34:30.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:34:30.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:34:30.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:34:30.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:30.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:34:30.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:34:30.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:30.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:34:30.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:34:30.970 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:30.970 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:30.970 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:30.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.970 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:30.970 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:30.970 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.072311012, 2024-04-24T14:34:30.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:34:30.971 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:30.971 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:30.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:30.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:30.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:30.971 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:34:30.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.971 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:30.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:30.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:30.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:30.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:30.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:30.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:30.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:30.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:30.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:34:30.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:34:30.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:30.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:34:30.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:34:30.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:34:30.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:34:30.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:34:30.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:34:30.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:30.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:34:30.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:34:30.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:34:30.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:34:30.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:30.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:34:30.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:34:30.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:34:30.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:34:30.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:30.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:34:30.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:34:30.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:30.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:34:30.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:34:30.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:30.977 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:30.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:30.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.977 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:30.977 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:30.977 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.072295230999998, 2024-04-24T14:34:30.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:34:30.978 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:30.978 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:30.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:30.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:30.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:30.978 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:34:30.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.978 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:30.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:30.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:30.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:30.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:30.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:30.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:30.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:30.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:30.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:34:30.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:34:30.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:30.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:34:30.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:34:30.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:34:30.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:34:30.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:34:30.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:34:30.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:30.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:34:30.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:34:30.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:34:30.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:34:30.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:30.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:34:30.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:34:30.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:34:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:34:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:34:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:34:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:30.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:34:30.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:34:30.984 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:30.984 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:30.984 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:30.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.984 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:30.984 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:30.984 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.072280942999999, 2024-04-24T14:34:30.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:34:30.984 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:30.985 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:30.985 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:30.985 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:30.985 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:30.985 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:34:30.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.985 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:30.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:30.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:30.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:30.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:30.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:30.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:30.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:30.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:30.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:34:30.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:34:30.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:30.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:34:30.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:34:30.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:34:30.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:34:30.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:34:30.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:34:30.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:30.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:34:30.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:34:30.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:34:30.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:34:30.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:30.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:34:30.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:34:30.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:34:30.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:34:30.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:30.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:34:30.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:34:30.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:30.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:34:30.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:34:30.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:30.991 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:30.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:30.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.991 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:30.991 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:30.991 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.072261190999999, 2024-04-24T14:34:30.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:34:30.991 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:30.991 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:30.992 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:30.992 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:30.992 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:30.992 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:34:30.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.992 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:30.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:30.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:30.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:30.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:30.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:30.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:30.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:30.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:30.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:34:30.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:34:30.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:30.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:34:30.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:34:30.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:30.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:34:30.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:34:30.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:34:30.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:34:30.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:30.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:34:30.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:34:30.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:34:30.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:34:30.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:30.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:34:30.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:34:30.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:30.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:34:30.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:34:30.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:30.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:34:30.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:34:30.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:30.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:34:30.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:34:30.997 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:30.997 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:30.997 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:30.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.998 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:30.998 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:30.998 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.072243744000001, 2024-04-24T14:34:30.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:34:30.998 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:30.998 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:30.998 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:30.998 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:30.998 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:30.999 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:34:30.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.999 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:30.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:30.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:30.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:30.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:30.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:30.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:30.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:31.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:31.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:34:31.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:34:31.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:34:31.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:34:31.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:34:31.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:34:31.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:34:31.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:34:31.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:31.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:34:31.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:34:31.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:34:31.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:34:31.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:31.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:34:31.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:34:31.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:34:31.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:34:31.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:34:31.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:34:31.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:34:31.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:34:31.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.005 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.005 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:100 3.2 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.005 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:31.005 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.1981889289999996, 2024-04-24T14:34:31.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009078015, 2024-04-24T14:34:31.005 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.005 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.006 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 100 2024-04-24T14:34:31.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.006 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:31.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:31.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754019+0000", 2024-04-24T14:34:31.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754025+0000", 2024-04-24T14:34:31.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2110000000000003e-06 2024-04-24T14:34:31.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754037+0000", 2024-04-24T14:34:31.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1868999999999999e-05 2024-04-24T14:34:31.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754046+0000", 2024-04-24T14:34:31.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7430000000000008e-06 2024-04-24T14:34:31.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754095+0000", 2024-04-24T14:34:31.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8924000000000001e-05 2024-04-24T14:34:31.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754153+0000", 2024-04-24T14:34:31.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8165000000000003e-05 2024-04-24T14:34:31.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:31.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754262+0000", 2024-04-24T14:34:31.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108694 2024-04-24T14:34:31.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.762999+0000", 2024-04-24T14:34:31.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087372590000000007 2024-04-24T14:34:31.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763027+0000", 2024-04-24T14:34:31.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8467e-05 2024-04-24T14:34:31.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763070+0000", 2024-04-24T14:34:31.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2236e-05 2024-04-24T14:34:31.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763098+0000", 2024-04-24T14:34:31.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8561999999999999e-05 2024-04-24T14:34:31.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.012 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.012 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:101 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [delete] snapc 7=[7,6,4] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.012 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:31.012 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9929812629999999, 2024-04-24T14:34:31.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012894743, 2024-04-24T14:34:31.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.013 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.013 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 101 2024-04-24T14:34:31.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.013 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:31.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:31.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959227+0000", 2024-04-24T14:34:31.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959233+0000", 2024-04-24T14:34:31.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.857e-06 2024-04-24T14:34:31.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959249+0000", 2024-04-24T14:34:31.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6155999999999999e-05 2024-04-24T14:34:31.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959257+0000", 2024-04-24T14:34:31.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7440000000000007e-06 2024-04-24T14:34:31.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959790+0000", 2024-04-24T14:34:31.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053381100000000003 2024-04-24T14:34:31.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960206+0000", 2024-04-24T14:34:31.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041549700000000002 2024-04-24T14:34:31.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:31.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960744+0000", 2024-04-24T14:34:31.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053762199999999997 2024-04-24T14:34:31.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971963+0000", 2024-04-24T14:34:31.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011219816000000001 2024-04-24T14:34:31.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972021+0000", 2024-04-24T14:34:31.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7855000000000001e-05 2024-04-24T14:34:31.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972073+0000", 2024-04-24T14:34:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2196e-05 2024-04-24T14:34:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972123+0000", 2024-04-24T14:34:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9311999999999998e-05 2024-04-24T14:34:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:102 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9929412370000001, 2024-04-24T14:34:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012939713, 2024-04-24T14:34:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 102 2024-04-24T14:34:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959267+0000", 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959274+0000", 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.593e-06 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959291+0000", 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6387000000000001e-05 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959301+0000", 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0023e-05 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969594+0000", 2024-04-24T14:34:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010293182 2024-04-24T14:34:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969902+0000", 2024-04-24T14:34:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030835299999999998 2024-04-24T14:34:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970408+0000", 2024-04-24T14:34:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050588100000000002 2024-04-24T14:34:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972009+0000", 2024-04-24T14:34:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016011210000000001 2024-04-24T14:34:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972132+0000", 2024-04-24T14:34:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122521 2024-04-24T14:34:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972174+0000", 2024-04-24T14:34:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2242e-05 2024-04-24T14:34:31.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972208+0000", 2024-04-24T14:34:31.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3522999999999997e-05 2024-04-24T14:34:31.028 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.028 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.028 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.028 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:104 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.028 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:31.028 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9923300230000001, 2024-04-24T14:34:31.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014589062, 2024-04-24T14:34:31.029 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.029 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.029 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.029 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.029 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.029 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 104 2024-04-24T14:34:31.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.029 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:31.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959877+0000", 2024-04-24T14:34:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959890+0000", 2024-04-24T14:34:31.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2537e-05 2024-04-24T14:34:31.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959909+0000", 2024-04-24T14:34:31.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9340000000000001e-05 2024-04-24T14:34:31.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959922+0000", 2024-04-24T14:34:31.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2882e-05 2024-04-24T14:34:31.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970900+0000", 2024-04-24T14:34:31.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010977736 2024-04-24T14:34:31.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971039+0000", 2024-04-24T14:34:31.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139593 2024-04-24T14:34:31.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:31.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971453+0000", 2024-04-24T14:34:31.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041351099999999998 2024-04-24T14:34:31.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974044+0000", 2024-04-24T14:34:31.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025908289999999998 2024-04-24T14:34:31.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974377+0000", 2024-04-24T14:34:31.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033310399999999997 2024-04-24T14:34:31.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974433+0000", 2024-04-24T14:34:31.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6166999999999999e-05 2024-04-24T14:34:31.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974468+0000", 2024-04-24T14:34:31.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5336000000000002e-05 2024-04-24T14:34:31.034 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.034 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.034 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.035 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:106 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.035 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:31.035 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9922706899999998, 2024-04-24T14:34:31.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017192064999999999, 2024-04-24T14:34:31.035 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.035 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.035 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 106 2024-04-24T14:34:31.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.035 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959938+0000", 2024-04-24T14:34:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959943+0000", 2024-04-24T14:34:31.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.677e-06 2024-04-24T14:34:31.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959953+0000", 2024-04-24T14:34:31.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8069999999999994e-06 2024-04-24T14:34:31.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:34:31.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0560000000000001e-06 2024-04-24T14:34:31.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972232+0000", 2024-04-24T14:34:31.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012272460000000001 2024-04-24T14:34:31.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972304+0000", 2024-04-24T14:34:31.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2630000000000001e-05 2024-04-24T14:34:31.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:31.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972548+0000", 2024-04-24T14:34:31.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000244047 2024-04-24T14:34:31.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976900+0000", 2024-04-24T14:34:31.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043514069999999998 2024-04-24T14:34:31.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977076+0000", 2024-04-24T14:34:31.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176281 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977106+0000", 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0015e-05 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977131+0000", 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4669e-05 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.041 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:114 3.2 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.041 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:31.041 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9689495259999998, 2024-04-24T14:34:31.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090880089999999993, 2024-04-24T14:34:31.041 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.041 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.041 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.041 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.041 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.041 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 114 2024-04-24T14:34:31.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.041 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:31.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:31.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983259+0000", 2024-04-24T14:34:31.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983264+0000", 2024-04-24T14:34:31.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9130000000000002e-06 2024-04-24T14:34:31.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983277+0000", 2024-04-24T14:34:31.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3723999999999999e-05 2024-04-24T14:34:31.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983282+0000", 2024-04-24T14:34:31.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0429999999999998e-06 2024-04-24T14:34:31.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983339+0000", 2024-04-24T14:34:31.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6323000000000003e-05 2024-04-24T14:34:31.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983464+0000", 2024-04-24T14:34:31.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012554299999999999 2024-04-24T14:34:31.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:31.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983693+0000", 2024-04-24T14:34:31.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000228848 2024-04-24T14:34:31.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.986978+0000", 2024-04-24T14:34:31.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032852810000000001 2024-04-24T14:34:31.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992288+0000", 2024-04-24T14:34:31.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053095369999999996 2024-04-24T14:34:31.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992322+0000", 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3683999999999998e-05 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992348+0000", 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6157999999999999e-05 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:121 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9568934269999998, 2024-04-24T14:34:31.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098798159999999996, 2024-04-24T14:34:31.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.047 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.047 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 121 2024-04-24T14:34:31.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.047 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:31.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:31.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995315+0000", 2024-04-24T14:34:31.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995320+0000", 2024-04-24T14:34:31.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0239999999999997e-06 2024-04-24T14:34:31.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995328+0000", 2024-04-24T14:34:31.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7800000000000001e-06 2024-04-24T14:34:31.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995334+0000", 2024-04-24T14:34:31.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2330000000000003e-06 2024-04-24T14:34:31.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998810+0000", 2024-04-24T14:34:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034756090000000002 2024-04-24T14:34:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998870+0000", 2024-04-24T14:34:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0625999999999999e-05 2024-04-24T14:34:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.999091+0000", 2024-04-24T14:34:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000220631 2024-04-24T14:34:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.001389+0000", 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022977549999999998 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005097+0000", 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037085439999999998 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005154+0000", 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6208000000000001e-05 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005196+0000", 2024-04-24T14:34:31.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2052000000000003e-05 2024-04-24T14:34:31.052 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.052 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.052 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.052 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:127 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.052 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:31.052 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9465919110000001, 2024-04-24T14:34:31.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013414918, 2024-04-24T14:34:31.052 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.052 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.052 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.053 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2024-04-24T14:34:31.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.053 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:31.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:31.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005616+0000", 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005622+0000", 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2020000000000003e-06 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005632+0000", 2024-04-24T14:34:31.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2024-04-24T14:34:31.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005637+0000", 2024-04-24T14:34:31.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7439999999999998e-06 2024-04-24T14:34:31.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005947+0000", 2024-04-24T14:34:31.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000310792 2024-04-24T14:34:31.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006059+0000", 2024-04-24T14:34:31.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011192399999999999 2024-04-24T14:34:31.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:31.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006503+0000", 2024-04-24T14:34:31.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000443394 2024-04-24T14:34:31.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018864+0000", 2024-04-24T14:34:31.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012361162 2024-04-24T14:34:31.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018877+0000", 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3497e-05 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018965+0000", 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7476000000000001e-05 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019032+0000", 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7469000000000001e-05 2024-04-24T14:34:31.057 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:129 3.2 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9464516779999999, 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010059754000000001, 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 129 2024-04-24T14:34:31.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.059 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:31.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:31.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005757+0000", 2024-04-24T14:34:31.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005760+0000", 2024-04-24T14:34:31.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2430000000000001e-06 2024-04-24T14:34:31.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005769+0000", 2024-04-24T14:34:31.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4810000000000006e-06 2024-04-24T14:34:31.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005773+0000", 2024-04-24T14:34:31.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:31.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005840+0000", 2024-04-24T14:34:31.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6934999999999995e-05 2024-04-24T14:34:31.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005961+0000", 2024-04-24T14:34:31.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012153 2024-04-24T14:34:31.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:31.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006181+0000", 2024-04-24T14:34:31.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021947500000000001 2024-04-24T14:34:31.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015520+0000", 2024-04-24T14:34:31.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093393049999999991 2024-04-24T14:34:31.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015733+0000", 2024-04-24T14:34:31.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021355699999999999 2024-04-24T14:34:31.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015773+0000", 2024-04-24T14:34:31.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9554000000000001e-05 2024-04-24T14:34:31.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015817+0000", 2024-04-24T14:34:31.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4325e-05 2024-04-24T14:34:31.063 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.063 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.063 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:131 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9461737970000001, 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014251976, 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006034+0000", 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006042+0000", 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2970000000000004e-06 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006061+0000", 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8505999999999999e-05 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006071+0000", 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0409e-05 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007899+0000", 2024-04-24T14:34:31.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018274789999999999 2024-04-24T14:34:31.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008281+0000", 2024-04-24T14:34:31.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038244700000000001 2024-04-24T14:34:31.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:31.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008528+0000", 2024-04-24T14:34:31.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024723900000000001 2024-04-24T14:34:31.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019070+0000", 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010541867999999999 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020099+0000", 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00102852 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020226+0000", 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012744300000000001 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020287+0000", 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1253000000000005e-05 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:132 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9461199439999999, 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015917981000000001, 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.069 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.070 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.070 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.070 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 132 2024-04-24T14:34:31.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.070 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:31.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006093+0000", 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3819999999999997e-06 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006103+0000", 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0352e-05 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006110+0000", 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3910000000000003e-06 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015586+0000", 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094759119999999995 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015730+0000", 2024-04-24T14:34:31.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014487199999999999 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015886+0000", 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015573799999999999 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020306+0000", 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044199399999999998 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021887+0000", 2024-04-24T14:34:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015809400000000001 2024-04-24T14:34:31.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021942+0000", 2024-04-24T14:34:31.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5071000000000002e-05 2024-04-24T14:34:31.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.022007+0000", 2024-04-24T14:34:31.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5110000000000005e-05 2024-04-24T14:34:31.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.074 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.075 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.5324.0:127 3.0 e87/69 ondisk, result = 0)", 2024-04-24T14:34:31.075 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:31.075 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.9444108760000001, 2024-04-24T14:34:31.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011245125, 2024-04-24T14:34:31.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.075 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:31.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:31.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:31.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007804+0000", 2024-04-24T14:34:31.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6610000000000003e-06 2024-04-24T14:34:31.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007817+0000", 2024-04-24T14:34:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3030999999999999e-05 2024-04-24T14:34:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007832+0000", 2024-04-24T14:34:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.47e-05 2024-04-24T14:34:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018843+0000", 2024-04-24T14:34:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011010647 2024-04-24T14:34:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018853+0000", 2024-04-24T14:34:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319000000000001e-05 2024-04-24T14:34:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019043+0000", 2024-04-24T14:34:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019055600000000001 2024-04-24T14:34:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.078 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.079 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.079 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:31.080 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=19389 2024-04-24T14:34:31.080 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 19389 2024-04-24T14:34:31.080 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 19389 -C rbd-nbd 2024-04-24T14:34:31.080 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:31.080 INFO:tasks.workunit.client.0.smithi173.stdout: 19389 ? 00:00:00 rbd-nbd 2024-04-24T14:34:31.081 INFO:tasks.workunit.client.0.smithi173.stderr:+ ls /tmp/tmp.waDYLkcsIV/mnt/ 2024-04-24T14:34:31.081 INFO:tasks.workunit.client.0.smithi173.stdout:lost+found 2024-04-24T14:34:31.081 INFO:tasks.workunit.client.0.smithi173.stdout:test 2024-04-24T14:34:31.082 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:31.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ dd if=/tmp/tmp.waDYLkcsIV/mnt/test of=/dev/null bs=1M count=1 2024-04-24T14:34:31.085 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records in 2024-04-24T14:34:31.085 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records out 2024-04-24T14:34:31.085 INFO:tasks.workunit.client.0.smithi173.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126728 s, 82.7 MB/s 2024-04-24T14:34:31.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo dd if=/tmp/tmp.waDYLkcsIV/data of=/tmp/tmp.waDYLkcsIV/mnt/test1 bs=1M count=1 oflag=direct 2024-04-24T14:34:31.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:31.085 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:31.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:31.085 INFO:tasks.workunit.client.0.smithi173.stderr:++ which dd 2024-04-24T14:34:31.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/dd 2024-04-24T14:34:31.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:31.085 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.waDYLkcsIV/data of=/tmp/tmp.waDYLkcsIV/mnt/test1 bs=1M count=1 oflag=direct 2024-04-24T14:34:31.085 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records in 2024-04-24T14:34:31.085 INFO:tasks.workunit.client.0.smithi173.stderr:1+0 records out 2024-04-24T14:34:31.086 INFO:tasks.workunit.client.0.smithi173.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263953 s, 39.7 MB/s 2024-04-24T14:34:31.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo umount /tmp/tmp.waDYLkcsIV/mnt 2024-04-24T14:34:31.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:31.086 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:31.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:31.086 INFO:tasks.workunit.client.0.smithi173.stderr:++ which umount 2024-04-24T14:34:31.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/umount 2024-04-24T14:34:31.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:31.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.waDYLkcsIV/mnt 2024-04-24T14:34:31.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device /dev/nbd0 19389 2024-04-24T14:34:31.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=/dev/nbd0 2024-04-24T14:34:31.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=19389 2024-04-24T14:34:31.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:34:31.086 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:31.087 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:31.087 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:31.087 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:31.087 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:31.087 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:31.087 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-04-24T14:34:31.088 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:31.085+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:31.158 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:34:31.158 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^19389\b' 2024-04-24T14:34:31.158 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^19389\b' 2024-04-24T14:34:31.180 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:31.182+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:31.189 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:31.189 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:31.189 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:31.189 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:31.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:31.211 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:31.213 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:34:31.213 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *19389\b' 2024-04-24T14:34:31.213 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *19389\b' 2024-04-24T14:34:31.226 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:31.226 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:34:31.281 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:31.282+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:31.295 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:31.295 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:31.295 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:31.295 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:31.295 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:31.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:31.382 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:31.385+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:31.404 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:31.404 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:31.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:31.405 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:31.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.405 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.405 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:31.405 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.651458402000003, 2024-04-24T14:34:31.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:34:31.405 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.405 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.405 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:34:31.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.406 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:31.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:31.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:34:31.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:34:31.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:34:31.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:34:31.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:34:31.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:34:31.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:34:31.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:34:31.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:34:31.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:34:31.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:34:31.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:34:31.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:34:31.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:34:31.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:34:31.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:34:31.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:34:31.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:34:31.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:34:31.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:34:31.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:34:31.411 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.411 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.411 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.411 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.411 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:31.411 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.651407568000003, 2024-04-24T14:34:31.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:34:31.411 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.412 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.412 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.412 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.412 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.412 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:34:31.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.412 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:31.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:31.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:31.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:34:31.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:34:31.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:34:31.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:34:31.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:34:31.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:34:31.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:34:31.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:34:31.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:34:31.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:34:31.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:34:31.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:34:31.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:34:31.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:34:31.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:34:31.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:34:31.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:34:31.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:34:31.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:34:31.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:34:31.417 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.417 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.417 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.418 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.418 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:31.418 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.651371261000001, 2024-04-24T14:34:31.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:34:31.418 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.418 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.418 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.418 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.418 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.418 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:34:31.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.419 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:31.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:31.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:34:31.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:34:31.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:34:31.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:34:31.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:34:31.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:34:31.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:34:31.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:34:31.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:34:31.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:34:31.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:34:31.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:34:31.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:34:31.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:34:31.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:34:31.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:34:31.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:34:31.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:34:31.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:34:31.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:34:31.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:34:31.424 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.424 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.424 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.424 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.424 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:31.424 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.647960011000002, 2024-04-24T14:34:31.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:34:31.424 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.424 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.425 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:34:31.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.425 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:31.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:31.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:34:31.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:31.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:34:31.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:34:31.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:34:31.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:34:31.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:34:31.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:34:31.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:34:31.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:34:31.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:34:31.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:34:31.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:34:31.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:34:31.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:34:31.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:34:31.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:34:31.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:34:31.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:34:31.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:34:31.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:34:31.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:34:31.430 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.430 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.430 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.430 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.430 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:31.430 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.647898333999997, 2024-04-24T14:34:31.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:34:31.433 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.434 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.434 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.434 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.434 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.434 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:34:31.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.434 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:31.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:31.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:34:31.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:34:31.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:34:31.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:34:31.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:34:31.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:34:31.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:34:31.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:34:31.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:34:31.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:34:31.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:34:31.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:34:31.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:34:31.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:34:31.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:34:31.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:31.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:34:31.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:34:31.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:34:31.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:34:31.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:34:31.440 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.440 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.440 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.440 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.440 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:31.440 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.647863758, 2024-04-24T14:34:31.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:34:31.440 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.441 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.441 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.441 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.441 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.441 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:34:31.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.441 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:31.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:31.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:31.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:34:31.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:34:31.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:34:31.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:34:31.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:34:31.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:34:31.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:34:31.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:34:31.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:34:31.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:34:31.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:34:31.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:34:31.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:34:31.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:34:31.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:34:31.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:34:31.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:34:31.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:34:31.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:34:31.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:34:31.446 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.447 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.447 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.447 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.447 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:31.447 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.647834418999999, 2024-04-24T14:34:31.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:34:31.447 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.447 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.447 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.447 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.447 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.448 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:34:31.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.448 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:31.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:31.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:34:31.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:34:31.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:34:31.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:34:31.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:34:31.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:34:31.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:34:31.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:34:31.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:34:31.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:34:31.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:34:31.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:34:31.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:34:31.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:34:31.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:34:31.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:34:31.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:34:31.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:34:31.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:34:31.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:34:31.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:34:31.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.453 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.454 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.454 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:31.454 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.647757837, 2024-04-24T14:34:31.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:34:31.454 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.454 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.454 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:34:31.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.455 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:31.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:31.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:34:31.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:34:31.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:34:31.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:34:31.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:34:31.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:34:31.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:34:31.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:34:31.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:34:31.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:34:31.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:34:31.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:34:31.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:34:31.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:34:31.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:34:31.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:34:31.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:34:31.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:34:31.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:34:31.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:34:31.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:34:31.460 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.460 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.460 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.461 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.461 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:31.461 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.647710291999999, 2024-04-24T14:34:31.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:34:31.461 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.461 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.461 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.461 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.461 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.461 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:34:31.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.461 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:31.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:31.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:34:31.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:34:31.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:34:31.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:34:31.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:34:31.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:34:31.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:34:31.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:34:31.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:34:31.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:34:31.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:34:31.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:34:31.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:34:31.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:34:31.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:34:31.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:34:31.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:34:31.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:34:31.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:34:31.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:34:31.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:34:31.467 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.467 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.467 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.467 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.467 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:31.467 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.647669649999997, 2024-04-24T14:34:31.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:34:31.468 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.468 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.468 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.468 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.468 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.468 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:34:31.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.468 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:31.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:31.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:31.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:34:31.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:34:31.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:34:31.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:34:31.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:34:31.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:34:31.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:34:31.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:34:31.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:34:31.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:34:31.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:34:31.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:34:31.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:34:31.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:34:31.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:34:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:34:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:34:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:34:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:34:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:34:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.647652538999999, 2024-04-24T14:34:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:34:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:34:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:34:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:34:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:34:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:34:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:34:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:34:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:34:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:34:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:34:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:34:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:34:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:34:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:34:31.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:34:31.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:34:31.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:34:31.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:34:31.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:34:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:34:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:34:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:34:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:103 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.4480864259999997, 2024-04-24T14:34:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032646351999999997, 2024-04-24T14:34:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 103 2024-04-24T14:34:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959674+0000", 2024-04-24T14:34:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959688+0000", 2024-04-24T14:34:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2024-04-24T14:34:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959722+0000", 2024-04-24T14:34:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3936000000000002e-05 2024-04-24T14:34:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959735+0000", 2024-04-24T14:34:31.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2926e-05 2024-04-24T14:34:31.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959921+0000", 2024-04-24T14:34:31.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185601 2024-04-24T14:34:31.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960324+0000", 2024-04-24T14:34:31.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040356700000000002 2024-04-24T14:34:31.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:31.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960839+0000", 2024-04-24T14:34:31.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051410599999999996 2024-04-24T14:34:31.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.975634+0000", 2024-04-24T14:34:31.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014795770999999999 2024-04-24T14:34:31.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992113+0000", 2024-04-24T14:34:31.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016478982999999999 2024-04-24T14:34:31.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992228+0000", 2024-04-24T14:34:31.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114295 2024-04-24T14:34:31.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992323+0000", 2024-04-24T14:34:31.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5344000000000001e-05 2024-04-24T14:34:31.487 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.487 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.487 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.487 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:105 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.487 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:31.487 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.447836648, 2024-04-24T14:34:31.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032581757000000003, 2024-04-24T14:34:31.487 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.487 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.487 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.487 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.488 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105 2024-04-24T14:34:31.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.488 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:31.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:31.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959925+0000", 2024-04-24T14:34:31.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:31.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959937+0000", 2024-04-24T14:34:31.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2391999999999999e-05 2024-04-24T14:34:31.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:34:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2535999999999999e-05 2024-04-24T14:34:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959970+0000", 2024-04-24T14:34:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1039999999999999e-05 2024-04-24T14:34:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969531+0000", 2024-04-24T14:34:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095603949999999993 2024-04-24T14:34:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969669+0000", 2024-04-24T14:34:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013806499999999999 2024-04-24T14:34:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970010+0000", 2024-04-24T14:34:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034062200000000001 2024-04-24T14:34:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976983+0000", 2024-04-24T14:34:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069735600000000002 2024-04-24T14:34:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992413+0000", 2024-04-24T14:34:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015429841999999999 2024-04-24T14:34:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992457+0000", 2024-04-24T14:34:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4169999999999999e-05 2024-04-24T14:34:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992508+0000", 2024-04-24T14:34:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0995000000000002e-05 2024-04-24T14:34:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:113 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:31.494 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.4276103569999998, 2024-04-24T14:34:31.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014793863000000001, 2024-04-24T14:34:31.494 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.494 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.494 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.494 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.494 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.494 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 113 2024-04-24T14:34:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980152+0000", 2024-04-24T14:34:31.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980156+0000", 2024-04-24T14:34:31.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.619e-06 2024-04-24T14:34:31.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980163+0000", 2024-04-24T14:34:31.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7070000000000003e-06 2024-04-24T14:34:31.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980173+0000", 2024-04-24T14:34:31.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0957e-05 2024-04-24T14:34:31.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983381+0000", 2024-04-24T14:34:31.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032079930000000001 2024-04-24T14:34:31.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983628+0000", 2024-04-24T14:34:31.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024665700000000001 2024-04-24T14:34:31.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:31.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983964+0000", 2024-04-24T14:34:31.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000336332 2024-04-24T14:34:31.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992694+0000", 2024-04-24T14:34:31.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087297009999999994 2024-04-24T14:34:31.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994878+0000", 2024-04-24T14:34:31.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021841439999999998 2024-04-24T14:34:31.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994917+0000", 2024-04-24T14:34:31.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8544999999999998e-05 2024-04-24T14:34:31.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994947+0000", 2024-04-24T14:34:31.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9705000000000001e-05 2024-04-24T14:34:31.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.500 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.500 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:130 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.500 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:31.500 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.4019169219999998, 2024-04-24T14:34:31.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014646289999999999, 2024-04-24T14:34:31.501 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.501 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.501 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.501 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.501 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.501 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 130 2024-04-24T14:34:31.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.501 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:31.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:31.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005844+0000", 2024-04-24T14:34:31.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:31.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005853+0000", 2024-04-24T14:34:31.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1319999999999994e-06 2024-04-24T14:34:31.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005870+0000", 2024-04-24T14:34:31.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7448999999999998e-05 2024-04-24T14:34:31.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005876+0000", 2024-04-24T14:34:31.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3319999999999997e-06 2024-04-24T14:34:31.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005913+0000", 2024-04-24T14:34:31.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6835999999999998e-05 2024-04-24T14:34:31.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006017+0000", 2024-04-24T14:34:31.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010358500000000001 2024-04-24T14:34:31.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:31.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006139+0000", 2024-04-24T14:34:31.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001227 2024-04-24T14:34:31.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.010194+0000", 2024-04-24T14:34:31.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040549380000000001 2024-04-24T14:34:31.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020365+0000", 2024-04-24T14:34:31.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010170983 2024-04-24T14:34:31.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020428+0000", 2024-04-24T14:34:31.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2457000000000002e-05 2024-04-24T14:34:31.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020492+0000", 2024-04-24T14:34:31.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4592999999999996e-05 2024-04-24T14:34:31.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.507 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.507 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:155 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1601536~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.507 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:31.507 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.3682561350000002, 2024-04-24T14:34:31.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015144391, 2024-04-24T14:34:31.507 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.507 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.507 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.508 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 155 2024-04-24T14:34:31.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.508 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:31.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:31.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039506+0000", 2024-04-24T14:34:31.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039766+0000", 2024-04-24T14:34:31.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025994400000000002 2024-04-24T14:34:31.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039780+0000", 2024-04-24T14:34:31.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3366e-05 2024-04-24T14:34:31.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039787+0000", 2024-04-24T14:34:31.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4359999999999996e-06 2024-04-24T14:34:31.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.050886+0000", 2024-04-24T14:34:31.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011098993 2024-04-24T14:34:31.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051026+0000", 2024-04-24T14:34:31.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001395 2024-04-24T14:34:31.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:31.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051211+0000", 2024-04-24T14:34:31.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018547099999999999 2024-04-24T14:34:31.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054490+0000", 2024-04-24T14:34:31.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032785499999999999 2024-04-24T14:34:31.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054505+0000", 2024-04-24T14:34:31.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5906999999999999e-05 2024-04-24T14:34:31.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054594+0000", 2024-04-24T14:34:31.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8817e-05 2024-04-24T14:34:31.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054651+0000", 2024-04-24T14:34:31.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7040999999999998e-05 2024-04-24T14:34:31.513 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.513 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.513 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.514 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:156 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1732608~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.514 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:31.514 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.3679654289999998, 2024-04-24T14:34:31.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016895194999999998, 2024-04-24T14:34:31.514 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.514 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.515 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 156 2024-04-24T14:34:31.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.515 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:31.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:31.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039797+0000", 2024-04-24T14:34:31.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040065+0000", 2024-04-24T14:34:31.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000268361 2024-04-24T14:34:31.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040086+0000", 2024-04-24T14:34:31.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1024000000000001e-05 2024-04-24T14:34:31.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040094+0000", 2024-04-24T14:34:31.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9379999999999992e-06 2024-04-24T14:34:31.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051936+0000", 2024-04-24T14:34:31.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011842518 2024-04-24T14:34:31.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052062+0000", 2024-04-24T14:34:31.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125482 2024-04-24T14:34:31.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:31.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052206+0000", 2024-04-24T14:34:31.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143953 2024-04-24T14:34:31.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054715+0000", 2024-04-24T14:34:31.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025088240000000002 2024-04-24T14:34:31.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056534+0000", 2024-04-24T14:34:31.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001819436 2024-04-24T14:34:31.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056610+0000", 2024-04-24T14:34:31.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5383000000000007e-05 2024-04-24T14:34:31.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056693+0000", 2024-04-24T14:34:31.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3283999999999997e-05 2024-04-24T14:34:31.520 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.520 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.520 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:157 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1863680~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.3676580500000002, 2024-04-24T14:34:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016763674999999999, 2024-04-24T14:34:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157 2024-04-24T14:34:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:31.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:31.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040104+0000", 2024-04-24T14:34:31.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040244+0000", 2024-04-24T14:34:31.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014035499999999999 2024-04-24T14:34:31.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040255+0000", 2024-04-24T14:34:31.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0135999999999999e-05 2024-04-24T14:34:31.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040260+0000", 2024-04-24T14:34:31.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7919999999999998e-06 2024-04-24T14:34:31.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053406+0000", 2024-04-24T14:34:31.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013146023999999999 2024-04-24T14:34:31.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053531+0000", 2024-04-24T14:34:31.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012471800000000001 2024-04-24T14:34:31.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:31.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053715+0000", 2024-04-24T14:34:31.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018395099999999999 2024-04-24T14:34:31.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056711+0000", 2024-04-24T14:34:31.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029956520000000001 2024-04-24T14:34:31.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056783+0000", 2024-04-24T14:34:31.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2010999999999999e-05 2024-04-24T14:34:31.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056831+0000", 2024-04-24T14:34:31.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8189999999999998e-05 2024-04-24T14:34:31.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056869+0000", 2024-04-24T14:34:31.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7812000000000003e-05 2024-04-24T14:34:31.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.527 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.527 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:158 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1994752~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.527 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:31.527 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.3674961230000005, 2024-04-24T14:34:31.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016845334999999999, 2024-04-24T14:34:31.528 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.528 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:31.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.528 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2024-04-24T14:34:31.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.528 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:31.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:31.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040266+0000", 2024-04-24T14:34:31.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040379+0000", 2024-04-24T14:34:31.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112728 2024-04-24T14:34:31.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040387+0000", 2024-04-24T14:34:31.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8499999999999994e-06 2024-04-24T14:34:31.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040391+0000", 2024-04-24T14:34:31.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2719999999999999e-06 2024-04-24T14:34:31.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054803+0000", 2024-04-24T14:34:31.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014411548 2024-04-24T14:34:31.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054982+0000", 2024-04-24T14:34:31.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017941699999999999 2024-04-24T14:34:31.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.055167+0000", 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018542100000000001 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056718+0000", 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015505670000000001 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057020+0000", 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030227500000000002 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:31.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057070+0000", 2024-04-24T14:34:31.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9611000000000003e-05 2024-04-24T14:34:31.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057112+0000", 2024-04-24T14:34:31.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2373999999999998e-05 2024-04-24T14:34:31.533 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.533 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.534 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:159 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:31.534 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:31.534 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.3620611350000003, 2024-04-24T14:34:31.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015930650000000001, 2024-04-24T14:34:31.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.534 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.534 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.534 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:31.534 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2024-04-24T14:34:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045701+0000", 2024-04-24T14:34:31.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:31.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045707+0000", 2024-04-24T14:34:31.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.562e-06 2024-04-24T14:34:31.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045728+0000", 2024-04-24T14:34:31.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1109999999999999e-05 2024-04-24T14:34:31.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045738+0000", 2024-04-24T14:34:31.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0016e-05 2024-04-24T14:34:31.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056183+0000", 2024-04-24T14:34:31.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010444836000000001 2024-04-24T14:34:31.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056470+0000", 2024-04-24T14:34:31.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028696899999999998 2024-04-24T14:34:31.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057160+0000", 2024-04-24T14:34:31.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069004699999999995 2024-04-24T14:34:31.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057291+0000", 2024-04-24T14:34:31.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013096200000000001 2024-04-24T14:34:31.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.061633+0000", 2024-04-24T14:34:31.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004341414 2024-04-24T14:34:31.539 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.539 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.539 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.539 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.539 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.540 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:31.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:31.542 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:31.542+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:31.642 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:31.646+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:31.646 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:31.646 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:31.646 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:31.646 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:31.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:31.727 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -n '' ']' 2024-04-24T14:34:31.727 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:34:31.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd snap ls rbd/testrbdnbd16809 2024-04-24T14:34:31.728 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '$2 == "snap" {print $1}' 2024-04-24T14:34:31.743 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:31.746+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:31.797 INFO:tasks.workunit.client.0.smithi173.stderr:+ SNAPID=4 2024-04-24T14:34:31.798 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:31.798 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:31.798 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:31.799 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd --options try-netlink,show-cookie map --snap-id 4 rbd/testrbdnbd16809 2024-04-24T14:34:31.799 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:34:31.799 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:34:31.801 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:34:31.801 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:34:31.803 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:34:31.803 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:34:31.803 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options try-netlink,show-cookie map --snap-id 4 rbd/testrbdnbd16809 2024-04-24T14:34:31.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:31.843 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:31.849+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:31.911 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:31.911 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:31.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:31.911 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:31.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.911 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.443575432000003, 2024-04-24T14:34:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:34:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:34:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:34:31.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:34:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:34:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:34:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:34:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:34:31.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:34:31.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:34:31.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:34:31.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:34:31.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:34:31.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:34:31.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:34:31.916 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.916 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.916 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.916 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.916 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:31.917 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.068218729000002, 2024-04-24T14:34:31.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:34:31.917 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.917 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.917 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.917 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.917 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.917 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:34:31.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.917 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:31.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:31.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:34:31.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:34:31.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:34:31.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:34:31.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:34:31.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:34:31.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:31.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:34:31.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:34:31.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:34:31.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:34:31.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:34:31.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:34:31.921 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.921 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.921 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.921 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.921 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:31.922 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.066790574999999, 2024-04-24T14:34:31.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:34:31.922 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.922 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.922 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.922 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.922 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.922 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:34:31.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.922 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:31.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:31.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:34:31.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:34:31.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:34:31.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:34:31.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:34:31.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:34:31.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:34:31.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:34:31.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:34:31.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:34:31.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:34:31.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:34:31.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:34:31.926 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.926 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.926 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.927 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.927 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:31.927 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.066763485999999, 2024-04-24T14:34:31.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:34:31.927 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.927 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.927 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:34:31.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.927 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:31.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:31.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:34:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:34:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:34:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:34:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:34:31.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:34:31.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:34:31.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:34:31.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:34:31.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:34:31.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:34:31.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:34:31.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:34:31.931 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.931 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.931 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.932 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.932 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:31.932 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.066739519000002, 2024-04-24T14:34:31.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:34:31.932 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.932 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.932 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.932 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.932 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.932 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:34:31.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.932 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:34:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:34:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:34:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:34:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:34:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:34:31.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:34:31.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:34:31.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:34:31.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:34:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:34:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:34:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:34:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.936 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.05275554, 2024-04-24T14:34:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:34:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:34:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:31.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:31.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:34:31.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:34:31.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:34:31.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:34:31.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:34:31.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:34:31.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:34:31.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:34:31.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:34:31.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:34:31.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:34:31.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:34:31.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:34:31.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:34:31.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:34:31.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:34:31.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:34:31.942 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.943 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.943 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.943 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.943 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:31.943 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.052639229, 2024-04-24T14:34:31.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:34:31.943 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.943 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.943 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.943 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.944 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.944 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:34:31.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.944 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:31.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:31.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:34:31.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:31.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:34:31.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:34:31.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:34:31.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:34:31.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:34:31.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:34:31.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:34:31.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:34:31.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:34:31.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:34:31.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:34:31.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:34:31.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:34:31.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:34:31.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:34:31.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:34:31.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:34:31.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:34:31.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:34:31.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:34:31.949 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.949 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.949 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.950 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.950 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:31.950 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.052591571000001, 2024-04-24T14:34:31.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:34:31.950 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.950 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.950 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.950 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.950 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.950 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:34:31.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.950 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:31.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:31.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:34:31.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:34:31.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:34:31.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:34:31.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:34:31.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:34:31.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:34:31.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:34:31.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:34:31.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:34:31.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:34:31.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:34:31.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:34:31.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:34:31.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:34:31.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:34:31.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:34:31.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:34:31.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:34:31.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:34:31.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:34:31.956 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.956 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.956 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.956 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.956 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:31.956 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.048554273000001, 2024-04-24T14:34:31.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:34:31.956 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.957 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.957 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.957 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.957 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.957 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:34:31.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.957 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:31.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:31.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:34:31.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:31.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:34:31.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:34:31.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:34:31.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:34:31.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:34:31.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:34:31.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:31.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:34:31.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:34:31.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:34:31.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:34:31.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:34:31.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:34:31.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:34:31.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:34:31.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:34:31.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:34:31.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:34:31.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:34:31.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:34:31.962 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.962 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.962 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.963 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.963 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:31.963 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.048490514000001, 2024-04-24T14:34:31.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:34:31.963 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.963 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.964 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:34:31.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.964 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:31.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:31.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:34:31.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:34:31.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:31.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:34:31.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:34:31.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:34:31.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:34:31.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:34:31.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:34:31.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:34:31.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:34:31.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:34:31.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:34:31.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:34:31.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:34:31.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:34:31.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:34:31.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:34:31.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:34:31.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:34:31.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:34:31.969 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.969 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.969 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.969 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.969 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:31.969 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.048465024000002, 2024-04-24T14:34:31.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:34:31.969 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.969 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.970 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.970 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.970 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.970 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:34:31.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.970 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:31.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:31.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:31.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:31.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:34:31.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:34:31.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:34:31.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:34:31.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:34:31.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:34:31.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:34:31.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:34:31.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:34:31.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:34:31.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:34:31.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:34:31.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:34:31.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:34:31.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:34:31.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:34:31.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:34:31.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:34:31.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:34:31.975 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.975 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.975 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.976 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.976 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:31.976 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.048441273999998, 2024-04-24T14:34:31.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:34:31.976 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.976 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.976 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.976 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.976 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.976 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:34:31.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.976 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:31.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:31.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:34:31.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:34:31.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:34:31.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:34:31.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:34:31.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:34:31.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:34:31.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:34:31.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:34:31.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:34:31.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:34:31.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:34:31.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:34:31.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:34:31.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:34:31.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:34:31.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:34:31.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:34:31.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:34:31.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:34:31.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:34:31.982 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.982 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.982 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.982 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.982 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:31.982 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.048402760999998, 2024-04-24T14:34:31.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:34:31.982 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.982 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.983 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.983 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.983 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.983 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:34:31.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.983 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:31.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:31.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:34:31.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:34:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:34:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:34:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:34:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:34:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:34:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:34:31.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:34:31.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:34:31.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:34:31.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:34:31.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:34:31.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:34:31.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:34:31.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:34:31.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:34:31.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:34:31.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:34:31.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:34:31.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:34:31.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.988 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.988 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.989 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:31.989 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.048374564, 2024-04-24T14:34:31.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:34:31.989 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.989 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.989 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:34:31.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.989 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:31.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:31.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:34:31.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:31.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:34:31.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:34:31.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:34:31.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:34:31.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:34:31.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:34:31.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:34:31.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:34:31.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:34:31.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:34:31.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:34:31.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:34:31.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:31.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:34:31.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:34:31.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:34:31.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:34:31.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:31.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:34:31.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:34:31.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:31.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:34:31.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:34:31.995 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.995 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:31.995 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:31.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.995 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:31.995 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:31.995 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.048353106999997, 2024-04-24T14:34:31.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:34:31.995 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:31.996 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:31.996 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:31.996 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:31.996 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:31.996 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:34:31.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.996 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:31.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:31.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:31.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:31.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:31.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:31.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:31.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:34:31.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:31.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:31.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:34:31.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:34:31.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:31.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:34:31.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:34:31.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:31.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:34:31.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:31.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:34:31.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:34:31.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:31.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:34:31.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:34:31.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:31.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:31.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:31.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:34:31.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:34:32.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:32.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:34:32.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:34:32.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:32.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:34:32.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:34:32.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:32.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:34:32.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:34:32.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:32.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:34:32.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:34:32.001 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:32.001 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:32.002 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:32.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.002 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:32.002 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:32.002 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.048334265000001, 2024-04-24T14:34:32.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:34:32.002 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:32.002 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:32.002 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:32.002 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:32.002 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:32.002 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:34:32.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.003 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:32.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:32.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:32.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:32.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:32.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:32.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:32.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:32.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:34:32.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:32.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:32.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:34:32.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:34:32.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:32.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:34:32.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:34:32.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:32.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:34:32.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:34:32.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:32.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:34:32.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:34:32.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:32.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:34:32.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:34:32.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:32.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:34:32.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:34:32.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:32.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:34:32.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:34:32.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:32.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:34:32.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:34:32.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:32.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:34:32.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:34:32.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:32.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:34:32.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:34:32.008 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:32.008 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:32.008 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:32.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.008 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:32.008 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:32.008 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.835502116000001, 2024-04-24T14:34:32.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:34:32.008 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:32.008 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:32.009 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:32.009 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:34:32.009 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:32.009 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:34:32.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.009 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:32.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:32.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:32.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:32.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:32.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:32.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:32.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:32.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:34:32.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:32.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:32.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:34:32.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:34:32.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:32.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:34:32.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:32.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:32.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:34:32.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:34:32.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:32.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:34:32.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:34:32.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:32.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:34:32.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:34:32.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:32.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:34:32.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:34:32.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:32.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:34:32.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:34:32.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:32.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:34:32.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:34:32.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:32.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:34:32.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:34:32.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:32.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:34:32.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:34:32.014 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:32.014 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:32.014 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:32.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.015 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:39 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1753088~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:32.015 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:32.015 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.89572969999999996, 2024-04-24T14:34:32.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01139394, 2024-04-24T14:34:32.015 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:32.015 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:32.015 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:32.015 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:32.015 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:32.015 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 39 2024-04-24T14:34:32.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.015 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:32.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:32.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:32.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:32.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:32.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:32.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:32.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:32.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018967+0000", 2024-04-24T14:34:32.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:32.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:32.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019242+0000", 2024-04-24T14:34:32.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027504500000000002 2024-04-24T14:34:32.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:32.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019274+0000", 2024-04-24T14:34:32.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2472999999999999e-05 2024-04-24T14:34:32.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:32.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019288+0000", 2024-04-24T14:34:32.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3283e-05 2024-04-24T14:34:32.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:32.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026753+0000", 2024-04-24T14:34:32.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074651120000000003 2024-04-24T14:34:32.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:32.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026971+0000", 2024-04-24T14:34:32.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021810800000000001 2024-04-24T14:34:32.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:32.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.027211+0000", 2024-04-24T14:34:32.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023975 2024-04-24T14:34:32.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:32.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030200+0000", 2024-04-24T14:34:32.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029890709999999998 2024-04-24T14:34:32.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:32.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030252+0000", 2024-04-24T14:34:32.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1926999999999999e-05 2024-04-24T14:34:32.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:32.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030312+0000", 2024-04-24T14:34:32.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0602e-05 2024-04-24T14:34:32.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:32.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030363+0000", 2024-04-24T14:34:32.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0342e-05 2024-04-24T14:34:32.020 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:32.020 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:32.021 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:32.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.021 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:40 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1884160~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:32.021 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:32.021 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.895393726, 2024-04-24T14:34:32.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011199064, 2024-04-24T14:34:32.021 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:32.021 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:32.021 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:32.021 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:32.021 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:32.022 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 40 2024-04-24T14:34:32.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.022 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:32.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:32.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:32.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:32.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:32.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:32.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:32.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:32.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019303+0000", 2024-04-24T14:34:32.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:32.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:32.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019729+0000", 2024-04-24T14:34:32.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042620899999999998 2024-04-24T14:34:32.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:32.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019769+0000", 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9847e-05 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019797+0000", 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8347000000000002e-05 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028299+0000", 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085010629999999997 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:32.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028385+0000", 2024-04-24T14:34:32.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6824999999999996e-05 2024-04-24T14:34:32.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:32.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028510+0000", 2024-04-24T14:34:32.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012483000000000001 2024-04-24T14:34:32.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:32.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030410+0000", 2024-04-24T14:34:32.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001899612 2024-04-24T14:34:32.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:32.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030416+0000", 2024-04-24T14:34:32.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7699999999999998e-06 2024-04-24T14:34:32.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:32.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030461+0000", 2024-04-24T14:34:32.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5157999999999997e-05 2024-04-24T14:34:32.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030504+0000", 2024-04-24T14:34:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2988999999999998e-05 2024-04-24T14:34:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:41 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 2015232~81920] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.89488501899999995, 2024-04-24T14:34:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012345257, 2024-04-24T14:34:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 41 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019812+0000", 2024-04-24T14:34:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019975+0000", 2024-04-24T14:34:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016325700000000001 2024-04-24T14:34:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020025+0000", 2024-04-24T14:34:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9484999999999998e-05 2024-04-24T14:34:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020035+0000", 2024-04-24T14:34:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0444e-05 2024-04-24T14:34:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029150+0000", 2024-04-24T14:34:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091150160000000001 2024-04-24T14:34:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029352+0000", 2024-04-24T14:34:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020166499999999999 2024-04-24T14:34:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029611+0000", 2024-04-24T14:34:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025949600000000001 2024-04-24T14:34:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030518+0000", 2024-04-24T14:34:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090633300000000005 2024-04-24T14:34:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032088+0000", 2024-04-24T14:34:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015703290000000001 2024-04-24T14:34:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032130+0000", 2024-04-24T14:34:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2273e-05 2024-04-24T14:34:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032159+0000", 2024-04-24T14:34:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8387000000000001e-05 2024-04-24T14:34:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:32.033 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:32.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:32.036 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:32.038+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:32.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ OUT=/dev/nbd0 2024-04-24T14:34:32.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ read DEV COOKIE 2024-04-24T14:34:32.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:34:32.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:32.037 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:32.037 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:32.037 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:32.051 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=19486 2024-04-24T14:34:32.051 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 19486 2024-04-24T14:34:32.051 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 19486 -C rbd-nbd 2024-04-24T14:34:32.057 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:32.057 INFO:tasks.workunit.client.0.smithi173.stdout: 19486 ? 00:00:00 rbd-nbd 2024-04-24T14:34:32.057 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device detach rbd/testrbdnbd16809 --snap-id 4 --device-type nbd 2024-04-24T14:34:32.057 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:32.057 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:32.059 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:32.059 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:32.061 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:32.061 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:32.062 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd16809 --snap-id 4 --device-type nbd 2024-04-24T14:34:32.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:34:32.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T14:34:32.135 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:32.141+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:32.143 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:32.143 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:32.144 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:32.144 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:32.153 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:32.163 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false get_pid rbd 2024-04-24T14:34:32.163 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:34:32.164 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:32.164 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:32.166 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:32.166 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:32.230 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:32.236 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:32.238+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:32.243 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID= 2024-04-24T14:34:32.243 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n '' 2024-04-24T14:34:32.243 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 1 2024-04-24T14:34:32.243 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:32.243 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd16809 --device-type nbd 2024-04-24T14:34:32.243 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd16809 --device-type nbd 2024-04-24T14:34:32.244 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:32.244 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:32.244 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:32.245 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:32.247 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:32.247 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:32.247 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd16809 --device-type nbd 2024-04-24T14:34:32.248 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:32.253 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:32.254 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:32.254 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:32.254 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:32.254 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:32.263 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:32.272 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:32.286 INFO:tasks.workunit.client.0.smithi173.stderr:rbd: could not validate attach request 2024-04-24T14:34:32.286 INFO:tasks.workunit.client.0.smithi173.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-04-24T14:34:32.286 INFO:tasks.workunit.client.0.smithi173.stderr:rbd: must specify --cookie or --force to proceed 2024-04-24T14:34:32.289 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:32.289 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -n '' ']' 2024-04-24T14:34:32.289 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd16809 --device-type nbd --force 2024-04-24T14:34:32.289 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:32.289 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:32.290 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:32.290 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:32.291 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:32.292 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:32.293 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:32.293 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd0 --snap-id 4 rbd/testrbdnbd16809 --device-type nbd --force 2024-04-24T14:34:32.311 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:32.329 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:32.332 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:32.332 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:32.332 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:32.332 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:32.332 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:32.332 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:32.337 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:32.341+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:32.372 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:32.372 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:32.381 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:32.430 INFO:tasks.workunit.client.0.smithi173.stdout:/dev/nbd0 2024-04-24T14:34:32.438 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:32.442+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:32.439 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:34:32.440 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:32.440 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:32.440 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:32.441 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:32.478 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:32.479 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:32.488 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:32.519 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=19551 2024-04-24T14:34:32.520 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 19551 2024-04-24T14:34:32.520 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 19551 -C rbd-nbd 2024-04-24T14:34:32.529 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:34:32.529 INFO:tasks.workunit.client.0.smithi173.stdout: 19551 ? 00:00:00 rbd-nbd 2024-04-24T14:34:32.529 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device detach /dev/nbd0 --device-type nbd 2024-04-24T14:34:32.530 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:34:32.530 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:34:32.530 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:34:32.531 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:34:32.532 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:34:32.532 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:34:32.532 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd0 --device-type nbd 2024-04-24T14:34:32.539 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:32.542+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:32.583 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:32.583 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:32.592 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:32.627 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false get_pid rbd 2024-04-24T14:34:32.627 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:34:32.627 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:34:32.627 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:34:32.628 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:34:32.628 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd0'\'']/id' 2024-04-24T14:34:32.639 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:32.645+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:32.689 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:32.689 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:32.698 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID= 2024-04-24T14:34:32.698 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n '' 2024-04-24T14:34:32.698 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 1 2024-04-24T14:34:32.698 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:34:32.698 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:34:32.698 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd config image set rbd/testrbdnbd16809 rbd_discard_granularity_bytes 4096 2024-04-24T14:34:32.700 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:32.740 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:32.742+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:32.805 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd feature enable rbd/testrbdnbd16809 journaling 2024-04-24T14:34:32.841 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:32.842+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:32.846 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:32.846 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:32.856 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:32.917 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:34:32.917 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:34:32.917 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:34:32.919 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:34:32.920 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:34:32.922 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:34:32.922 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:34:32.922 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:34:32.941 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:32.946+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:32.955 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:32.955 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:32.964 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:33.042 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:33.045+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:33.060 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:33.061 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:33.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:33.121 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-24T14:34:33.142 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:33.146+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:33.181 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:33.181 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:33.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:33.243 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:33.249+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:33.285 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:33.285 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:33.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:33.344 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:33.346+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:33.386 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:33.386 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:33.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:33.446 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:33.450+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:33.496 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:33.496 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:33.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:33.546 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:33.549+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:33.604 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:33.604 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:33.614 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:33.647 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:33.650+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:33.717 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:33.717 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:33.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:33.748 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:33.753+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:33.849 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:33.850+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:33.874 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:33.874 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:33.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:33.949 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:33.953+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:33.984 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:33.985 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:33.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:34.050 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:34.053+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:34.089 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:34.090 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:34.099 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:34.103 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 1] 2024-04-24T14:34:34.103 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:34:34.103 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-24T14:34:34.103 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.484353 2024-04-24T14:34:34.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 0.48435285148068763 2024-04-24T14:34:34.151 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:34.157+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:34.208 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:34.208 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:34.217 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:34.251 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:34.254+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:34.314 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:34.314 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:34.324 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:34.342 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:34:34.386 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:34.390+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:34.422 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:34.423 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:34.432 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:34.452 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:34.454+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:34.474 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:34.531 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:34.531 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:34.540 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:34.554 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:34.558+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:34.639 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:34.639 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:34.648 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:34.655 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:34.658+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:34.707 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:34.733 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:34.751 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:34.751 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:34.756 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:34.758+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:34.758 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:34.760 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:34.778 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:34.797 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:34.799 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:34.799 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:34.799 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:34.800 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:34.800 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:34.800 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:34.856 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:34.857 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:34.857 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:34.857 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:34.857 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:34.858+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:34.865 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:34.958 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:34.961+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:34.961 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:34.961 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:34.961 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:34.961 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:34.961 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:34.971 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:35.059 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:35.061+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:35.068 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:35.068 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:35.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:35.068 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:35.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.068 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:34:35.068 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:35.068 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.001366060999999, 2024-04-24T14:34:35.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:34:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:34:35.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:35.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:34:35.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:34:35.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:34:35.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:34:35.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:34:35.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:34:35.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:34:35.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:34:35.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:34:35.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:34:35.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:35.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:34:35.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:34:35.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:34:35.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:34:35.072 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.073 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.073 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.073 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:34:35.073 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:35.073 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.006551481999999, 2024-04-24T14:34:35.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:34:35.073 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.073 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.073 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:35.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:35.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:34:35.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:35.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:34:35.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:34:35.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:34:35.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:34:35.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:34:35.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:34:35.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:34:35.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:34:35.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:34:35.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:34:35.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:35.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:34:35.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:34:35.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:34:35.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:34:35.081 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.081 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.082 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.082 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:34:35.082 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:35.082 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 40.995622224000002, 2024-04-24T14:34:35.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:34:35.082 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.082 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.082 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:35.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:35.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:34:35.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:35.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:34:35.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:34:35.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:34:35.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:34:35.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:34:35.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:34:35.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:34:35.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:34:35.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:34:35.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:34:35.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:34:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:34:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:34:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:34:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.086 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:34:35.087 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:35.087 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 40.984666142999998, 2024-04-24T14:34:35.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:34:35.087 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.087 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.087 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:35.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:35.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:34:35.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:35.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:34:35.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:34:35.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:34:35.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:34:35.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:34:35.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:34:35.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:34:35.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:34:35.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:34:35.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:34:35.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:35.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:34:35.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:34:35.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 40.973096777999999, 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.091 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:35.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:35.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:34:35.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:35.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:34:35.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:34:35.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:34:35.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:34:35.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:34:35.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:34:35.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:34:35.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:34:35.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:34:35.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:34:35.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:35.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:34:35.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:34:35.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:34:35.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:34:35.095 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.095 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.095 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 40.719977118999999, 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:35.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:34:35.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:35.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:34:35.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:34:35.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:34:35.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:34:35.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:34:35.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:34:35.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:34:35.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:34:35.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.099 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 40.709231951, 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:35.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:34:35.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:35.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:34:35.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:34:35.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:34:35.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:34:35.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:34:35.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:34:35.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:34:35.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:34:35.104 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.104 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.104 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.104 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:34:35.104 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:35.104 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 40.696865674999998, 2024-04-24T14:34:35.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:34:35.105 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.105 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.105 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:35.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:35.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:34:35.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:35.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:34:35.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:34:35.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:34:35.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:34:35.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:35.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:34:35.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:34:35.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:34:35.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:34:35.108 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.108 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.109 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.109 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:34:35.109 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:35.109 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 40.571929554, 2024-04-24T14:34:35.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:34:35.109 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.109 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.109 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:35.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:35.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:34:35.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:35.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:34:35.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:34:35.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:34:35.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:34:35.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:34:35.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:34:35.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:34:35.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:34:35.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:34:35.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:34:35.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:35.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:34:35.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:34:35.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.113 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:34:35.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:34:35.113 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.113 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.113 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.113 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:34:35.113 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:35.114 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 40.552541711000003, 2024-04-24T14:34:35.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:34:35.114 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.114 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.114 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:35.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:35.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:34:35.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:35.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:34:35.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:34:35.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:34:35.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:34:35.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:34:35.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:34:35.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:34:35.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:34:35.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:34:35.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:34:35.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:35.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:34:35.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:34:35.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:34:35.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:34:35.117 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.118 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.118 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.118 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:34:35.118 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:35.118 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 40.542813868000003, 2024-04-24T14:34:35.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:34:35.118 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.118 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.118 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:35.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:35.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:34:35.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:35.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:34:35.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:34:35.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:34:35.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:34:35.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:34:35.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:34:35.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:34:35.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:34:35.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:34:35.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:34:35.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:35.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 40.382704187999998, 2024-04-24T14:34:35.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:34:35.123 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.123 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.123 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:35.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:35.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:34:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:34:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:34:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:34:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:34:35.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:34:35.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:34:35.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:34:35.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:34:35.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:34:35.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:34:35.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:35.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:34:35.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:34:35.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:34:35.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:34:35.126 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.126 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.126 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.127 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:34:35.127 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:35.127 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 40.372975191999998, 2024-04-24T14:34:35.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:34:35.127 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.127 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.127 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:35.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:35.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:34:35.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:35.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:34:35.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:34:35.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:34:35.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:34:35.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:35.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:34:35.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:34:35.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.038004356999998, 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.131 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:35.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:35.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:35.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:35.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:34:35.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:34:35.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:34:35.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:34:35.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:34:35.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:34:35.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:34:35.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:34:35.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:34:35.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:34:35.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:35.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:34:35.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:34:35.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:34:35.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:34:35.135 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.135 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.135 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.135 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.193853609000001, 2024-04-24T14:34:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:34:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:34:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:34:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:34:35.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:34:35.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:34:35.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:34:35.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:34:35.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:34:35.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:34:35.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:34:35.139 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.139 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.139 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.139 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:5 3.2 3.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:34:35.139 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:35.139 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.865615967, 2024-04-24T14:34:35.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043717790000000001, 2024-04-24T14:34:35.139 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-24T14:34:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:34:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:34:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 5 2024-04-24T14:34:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207678+0000", 2024-04-24T14:34:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999962 2024-04-24T14:34:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207700+0000", 2024-04-24T14:34:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1313000000000001e-05 2024-04-24T14:34:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207757+0000", 2024-04-24T14:34:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7499e-05 2024-04-24T14:34:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207779+0000", 2024-04-24T14:34:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1478999999999999e-05 2024-04-24T14:34:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.212054+0000", 2024-04-24T14:34:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042750849999999996 2024-04-24T14:34:35.142 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.143 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.143 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.143 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:35.143 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:35.143 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.970090609, 2024-04-24T14:34:35.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044084859999999997, 2024-04-24T14:34:35.143 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.143 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:35.143 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103206+0000", 2024-04-24T14:34:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104731+0000", 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001524755 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104759+0000", 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7973000000000001e-05 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104807+0000", 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8112000000000003e-05 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104907+0000", 2024-04-24T14:34:35.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100317 2024-04-24T14:34:35.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104918+0000", 2024-04-24T14:34:35.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747999999999999e-05 2024-04-24T14:34:35.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.107616+0000", 2024-04-24T14:34:35.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026977559999999999 2024-04-24T14:34:35.147 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.147 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.147 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.147 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(3.2 83/82 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:35.147 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:35.147 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.963696232, 2024-04-24T14:34:35.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041425760000000002, 2024-04-24T14:34:35.147 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.147 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:35.147 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:35.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:35.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109600+0000", 2024-04-24T14:34:35.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:35.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111210+0000", 2024-04-24T14:34:35.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016097279999999999 2024-04-24T14:34:35.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111237+0000", 2024-04-24T14:34:35.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7189999999999999e-05 2024-04-24T14:34:35.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111251+0000", 2024-04-24T14:34:35.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.365e-05 2024-04-24T14:34:35.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111367+0000", 2024-04-24T14:34:35.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000115703 2024-04-24T14:34:35.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111378+0000", 2024-04-24T14:34:35.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1873e-05 2024-04-24T14:34:35.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.113744+0000", 2024-04-24T14:34:35.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023657069999999999 2024-04-24T14:34:35.151 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.151 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.151 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.151 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:35.151 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:35.151 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.946795655999999, 2024-04-24T14:34:35.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046193859999999996, 2024-04-24T14:34:35.151 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.151 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:35.151 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:35.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:35.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126501+0000", 2024-04-24T14:34:35.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:35.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128169+0000", 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016686229999999999 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128202+0000", 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2539000000000001e-05 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128222+0000", 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0149e-05 2024-04-24T14:34:35.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128375+0000", 2024-04-24T14:34:35.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000153459 2024-04-24T14:34:35.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128399+0000", 2024-04-24T14:34:35.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3393000000000001e-05 2024-04-24T14:34:35.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.131122+0000", 2024-04-24T14:34:35.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027227140000000002 2024-04-24T14:34:35.155 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.155 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.155 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.155 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 49'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@49'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:35.155 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:35.155 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.925014275999999, 2024-04-24T14:34:35.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038414529999999999, 2024-04-24T14:34:35.155 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:35.155 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:35.155 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:35.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148282+0000", 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:35.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149875+0000", 2024-04-24T14:34:35.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015929900000000001 2024-04-24T14:34:35.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:35.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149908+0000", 2024-04-24T14:34:35.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3102000000000003e-05 2024-04-24T14:34:35.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149941+0000", 2024-04-24T14:34:35.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2480000000000001e-05 2024-04-24T14:34:35.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:35.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150031+0000", 2024-04-24T14:34:35.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0394000000000002e-05 2024-04-24T14:34:35.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:35.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150043+0000", 2024-04-24T14:34:35.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1967000000000001e-05 2024-04-24T14:34:35.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:35.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:35.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:35.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.152125+0000", 2024-04-24T14:34:35.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020817029999999999 2024-04-24T14:34:35.159 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.159 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.159 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.159 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:35.159 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:35.159 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:35.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T14:34:35.161 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.0 primary-affinity to 0.484353 (8317422) 2024-04-24T14:34:35.162 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:35.162 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:35.165+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:35.262 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:35.265+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:35.269 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:35.269 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:35.269 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:35.269 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:35.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:35.364 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:35.369+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:35.377 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:35.377 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:35.386 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:35.465 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:35.469+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:35.485 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:35.485 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:35.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.608859471999999, 2024-04-24T14:34:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:34:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:34:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:34:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:34:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:34:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:34:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:34:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:34:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:34:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:34:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:34:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:34:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:34:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:34:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:34:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:34:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:34:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:34:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:34:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:34:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:34:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:34:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:34:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.608830486999999, 2024-04-24T14:34:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:34:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:34:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:34:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:34:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:34:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:34:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:34:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:34:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:34:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:34:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:34:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:34:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:34:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:34:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:34:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:34:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:34:35.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:34:35.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:34:35.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:34:35.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:34:35.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:34:35.497 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.497 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.497 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.498 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:35.498 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:35.498 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.608812002000001, 2024-04-24T14:34:35.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:34:35.498 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.498 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:35.498 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.498 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:35.498 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:35.498 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:34:35.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.498 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:35.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:35.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:34:35.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:35.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:34:35.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:34:35.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:34:35.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:34:35.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:34:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:34:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:34:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:34:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:34:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:34:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:34:35.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:34:35.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:35.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:34:35.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:34:35.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:34:35.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:34:35.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:34:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:34:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:34:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:34:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.608796720999997, 2024-04-24T14:34:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:34:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:34:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:34:35.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:34:35.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:34:35.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:34:35.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:34:35.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:34:35.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:35.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:34:35.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:34:35.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:34:35.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:34:35.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:34:35.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:34:35.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:34:35.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:34:35.509 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.509 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.509 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.509 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:35.510 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:35.510 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.608780940000003, 2024-04-24T14:34:35.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:34:35.510 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.510 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:35.510 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.510 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:35.510 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:35.510 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:34:35.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.510 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:35.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:35.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:35.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:35.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:34:35.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:34:35.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:34:35.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:34:35.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:34:35.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:34:35.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:34:35.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:34:35.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:35.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:34:35.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:34:35.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:34:35.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:34:35.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:35.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:34:35.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:34:35.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:34:35.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:34:35.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:34:35.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:34:35.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:34:35.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:34:35.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.515 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.515 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:35.515 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:35.515 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.608766652, 2024-04-24T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:35.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:35.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:35.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:34:35.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:34:35.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:34:35.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:34:35.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:34:35.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:34:35.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:34:35.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:34:35.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:35.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:34:35.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:34:35.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:34:35.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:34:35.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:35.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:34:35.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:34:35.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:34:35.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:34:35.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:34:35.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:34:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:34:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:34:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.6087469, 2024-04-24T14:34:35.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:34:35.522 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.522 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:35.522 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.522 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:35.522 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:35.522 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:34:35.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.522 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:35.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:35.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:35.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:35.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:34:35.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:34:35.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:34:35.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:34:35.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:34:35.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:34:35.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:34:35.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:34:35.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:35.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:34:35.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:34:35.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:34:35.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:34:35.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:35.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:34:35.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:34:35.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:34:35.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:34:35.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:34:35.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:34:35.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:34:35.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:34:35.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.527 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.527 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:35.527 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:35.528 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.608729453000002, 2024-04-24T14:34:35.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:34:35.528 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.528 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:35.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:35.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:35.528 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:34:35.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.528 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:35.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:35.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:35.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:35.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:34:35.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:34:35.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:34:35.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:34:35.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:34:35.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:34:35.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:34:35.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:34:35.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:35.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:34:35.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:34:35.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:34:35.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:34:35.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:35.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:34:35.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:34:35.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:34:35.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:34:35.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:34:35.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:34:35.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:34:35.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:34:35.533 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.533 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.533 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.533 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:100 3.2 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:35.534 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:35.534 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.7346746379999995, 2024-04-24T14:34:35.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009078015, 2024-04-24T14:34:35.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.534 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.534 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.534 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:35.534 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:35.534 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 100 2024-04-24T14:34:35.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.534 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:35.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:35.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754019+0000", 2024-04-24T14:34:35.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:35.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754025+0000", 2024-04-24T14:34:35.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2110000000000003e-06 2024-04-24T14:34:35.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754037+0000", 2024-04-24T14:34:35.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1868999999999999e-05 2024-04-24T14:34:35.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754046+0000", 2024-04-24T14:34:35.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7430000000000008e-06 2024-04-24T14:34:35.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754095+0000", 2024-04-24T14:34:35.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8924000000000001e-05 2024-04-24T14:34:35.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754153+0000", 2024-04-24T14:34:35.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8165000000000003e-05 2024-04-24T14:34:35.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:35.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754262+0000", 2024-04-24T14:34:35.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108694 2024-04-24T14:34:35.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:35.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.762999+0000", 2024-04-24T14:34:35.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087372590000000007 2024-04-24T14:34:35.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:35.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763027+0000", 2024-04-24T14:34:35.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8467e-05 2024-04-24T14:34:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763070+0000", 2024-04-24T14:34:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2236e-05 2024-04-24T14:34:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763098+0000", 2024-04-24T14:34:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8561999999999999e-05 2024-04-24T14:34:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:101 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [delete] snapc 7=[7,6,4] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.5294669719999998, 2024-04-24T14:34:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012894743, 2024-04-24T14:34:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 101 2024-04-24T14:34:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:35.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:35.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959227+0000", 2024-04-24T14:34:35.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:35.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959233+0000", 2024-04-24T14:34:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.857e-06 2024-04-24T14:34:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959249+0000", 2024-04-24T14:34:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6155999999999999e-05 2024-04-24T14:34:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959257+0000", 2024-04-24T14:34:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7440000000000007e-06 2024-04-24T14:34:35.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959790+0000", 2024-04-24T14:34:35.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053381100000000003 2024-04-24T14:34:35.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960206+0000", 2024-04-24T14:34:35.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041549700000000002 2024-04-24T14:34:35.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:35.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960744+0000", 2024-04-24T14:34:35.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053762199999999997 2024-04-24T14:34:35.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:35.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971963+0000", 2024-04-24T14:34:35.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011219816000000001 2024-04-24T14:34:35.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:35.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972021+0000", 2024-04-24T14:34:35.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7855000000000001e-05 2024-04-24T14:34:35.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:35.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972073+0000", 2024-04-24T14:34:35.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2196e-05 2024-04-24T14:34:35.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972123+0000", 2024-04-24T14:34:35.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9311999999999998e-05 2024-04-24T14:34:35.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.545 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.545 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:102 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:35.545 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:35.546 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.5294269459999992, 2024-04-24T14:34:35.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012939713, 2024-04-24T14:34:35.546 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.546 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:35.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:35.546 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 102 2024-04-24T14:34:35.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.546 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:35.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:35.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959267+0000", 2024-04-24T14:34:35.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:35.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959274+0000", 2024-04-24T14:34:35.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.593e-06 2024-04-24T14:34:35.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959291+0000", 2024-04-24T14:34:35.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6387000000000001e-05 2024-04-24T14:34:35.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959301+0000", 2024-04-24T14:34:35.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0023e-05 2024-04-24T14:34:35.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969594+0000", 2024-04-24T14:34:35.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010293182 2024-04-24T14:34:35.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969902+0000", 2024-04-24T14:34:35.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030835299999999998 2024-04-24T14:34:35.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:35.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970408+0000", 2024-04-24T14:34:35.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050588100000000002 2024-04-24T14:34:35.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:35.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972009+0000", 2024-04-24T14:34:35.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016011210000000001 2024-04-24T14:34:35.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:35.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972132+0000", 2024-04-24T14:34:35.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122521 2024-04-24T14:34:35.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:35.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972174+0000", 2024-04-24T14:34:35.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2242e-05 2024-04-24T14:34:35.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972208+0000", 2024-04-24T14:34:35.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3522999999999997e-05 2024-04-24T14:34:35.551 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.551 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.551 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.551 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:104 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:35.551 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:35.551 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.528815732, 2024-04-24T14:34:35.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014589062, 2024-04-24T14:34:35.552 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.552 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:35.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:35.552 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 104 2024-04-24T14:34:35.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.552 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:35.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959877+0000", 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959890+0000", 2024-04-24T14:34:35.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2537e-05 2024-04-24T14:34:35.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959909+0000", 2024-04-24T14:34:35.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9340000000000001e-05 2024-04-24T14:34:35.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959922+0000", 2024-04-24T14:34:35.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2882e-05 2024-04-24T14:34:35.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970900+0000", 2024-04-24T14:34:35.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010977736 2024-04-24T14:34:35.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971039+0000", 2024-04-24T14:34:35.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139593 2024-04-24T14:34:35.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:35.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971453+0000", 2024-04-24T14:34:35.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041351099999999998 2024-04-24T14:34:35.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:35.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974044+0000", 2024-04-24T14:34:35.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025908289999999998 2024-04-24T14:34:35.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:35.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974377+0000", 2024-04-24T14:34:35.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033310399999999997 2024-04-24T14:34:35.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:35.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974433+0000", 2024-04-24T14:34:35.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6166999999999999e-05 2024-04-24T14:34:35.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974468+0000", 2024-04-24T14:34:35.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5336000000000002e-05 2024-04-24T14:34:35.557 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.557 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.557 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.557 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:106 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:35.558 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:35.558 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.5287563990000006, 2024-04-24T14:34:35.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017192064999999999, 2024-04-24T14:34:35.558 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.558 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.558 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.558 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:35.558 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:35.558 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 106 2024-04-24T14:34:35.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.558 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:35.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:35.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959938+0000", 2024-04-24T14:34:35.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:35.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959943+0000", 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.677e-06 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959953+0000", 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8069999999999994e-06 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0560000000000001e-06 2024-04-24T14:34:35.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972232+0000", 2024-04-24T14:34:35.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012272460000000001 2024-04-24T14:34:35.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972304+0000", 2024-04-24T14:34:35.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2630000000000001e-05 2024-04-24T14:34:35.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:35.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972548+0000", 2024-04-24T14:34:35.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000244047 2024-04-24T14:34:35.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:35.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976900+0000", 2024-04-24T14:34:35.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043514069999999998 2024-04-24T14:34:35.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:35.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977076+0000", 2024-04-24T14:34:35.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176281 2024-04-24T14:34:35.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:35.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977106+0000", 2024-04-24T14:34:35.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0015e-05 2024-04-24T14:34:35.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977131+0000", 2024-04-24T14:34:35.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4669e-05 2024-04-24T14:34:35.563 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.563 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.563 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.563 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:114 3.2 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:35.563 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:35.564 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.5054352350000002, 2024-04-24T14:34:35.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090880089999999993, 2024-04-24T14:34:35.564 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.564 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.564 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.564 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:35.564 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:35.564 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 114 2024-04-24T14:34:35.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.564 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:35.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983259+0000", 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983264+0000", 2024-04-24T14:34:35.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9130000000000002e-06 2024-04-24T14:34:35.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983277+0000", 2024-04-24T14:34:35.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3723999999999999e-05 2024-04-24T14:34:35.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983282+0000", 2024-04-24T14:34:35.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0429999999999998e-06 2024-04-24T14:34:35.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983339+0000", 2024-04-24T14:34:35.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6323000000000003e-05 2024-04-24T14:34:35.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983464+0000", 2024-04-24T14:34:35.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012554299999999999 2024-04-24T14:34:35.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:35.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983693+0000", 2024-04-24T14:34:35.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000228848 2024-04-24T14:34:35.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:35.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.986978+0000", 2024-04-24T14:34:35.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032852810000000001 2024-04-24T14:34:35.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:35.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992288+0000", 2024-04-24T14:34:35.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053095369999999996 2024-04-24T14:34:35.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:35.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992322+0000", 2024-04-24T14:34:35.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3683999999999998e-05 2024-04-24T14:34:35.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992348+0000", 2024-04-24T14:34:35.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6157999999999999e-05 2024-04-24T14:34:35.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.569 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.569 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:121 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:35.569 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:35.569 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.4933791359999997, 2024-04-24T14:34:35.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098798159999999996, 2024-04-24T14:34:35.569 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.570 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:35.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:35.570 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 121 2024-04-24T14:34:35.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.570 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:35.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995315+0000", 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995320+0000", 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0239999999999997e-06 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995328+0000", 2024-04-24T14:34:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7800000000000001e-06 2024-04-24T14:34:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995334+0000", 2024-04-24T14:34:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2330000000000003e-06 2024-04-24T14:34:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998810+0000", 2024-04-24T14:34:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034756090000000002 2024-04-24T14:34:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998870+0000", 2024-04-24T14:34:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0625999999999999e-05 2024-04-24T14:34:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.999091+0000", 2024-04-24T14:34:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000220631 2024-04-24T14:34:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.001389+0000", 2024-04-24T14:34:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022977549999999998 2024-04-24T14:34:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:35.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005097+0000", 2024-04-24T14:34:35.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037085439999999998 2024-04-24T14:34:35.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:35.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005154+0000", 2024-04-24T14:34:35.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6208000000000001e-05 2024-04-24T14:34:35.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005196+0000", 2024-04-24T14:34:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2052000000000003e-05 2024-04-24T14:34:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:127 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.4830776199999995, 2024-04-24T14:34:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013414918, 2024-04-24T14:34:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2024-04-24T14:34:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005616+0000", 2024-04-24T14:34:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005622+0000", 2024-04-24T14:34:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2020000000000003e-06 2024-04-24T14:34:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005632+0000", 2024-04-24T14:34:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2024-04-24T14:34:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005637+0000", 2024-04-24T14:34:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7439999999999998e-06 2024-04-24T14:34:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005947+0000", 2024-04-24T14:34:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000310792 2024-04-24T14:34:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006059+0000", 2024-04-24T14:34:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011192399999999999 2024-04-24T14:34:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006503+0000", 2024-04-24T14:34:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000443394 2024-04-24T14:34:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018864+0000", 2024-04-24T14:34:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012361162 2024-04-24T14:34:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:35.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018877+0000", 2024-04-24T14:34:35.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3497e-05 2024-04-24T14:34:35.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:35.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018965+0000", 2024-04-24T14:34:35.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7476000000000001e-05 2024-04-24T14:34:35.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019032+0000", 2024-04-24T14:34:35.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7469000000000001e-05 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:129 3.2 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.4829373869999998, 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010059754000000001, 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:35.581 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 129 2024-04-24T14:34:35.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.582 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:35.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:35.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005757+0000", 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005760+0000", 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2430000000000001e-06 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005769+0000", 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4810000000000006e-06 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005773+0000", 2024-04-24T14:34:35.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:35.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005840+0000", 2024-04-24T14:34:35.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6934999999999995e-05 2024-04-24T14:34:35.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005961+0000", 2024-04-24T14:34:35.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012153 2024-04-24T14:34:35.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:35.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006181+0000", 2024-04-24T14:34:35.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021947500000000001 2024-04-24T14:34:35.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:35.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015520+0000", 2024-04-24T14:34:35.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093393049999999991 2024-04-24T14:34:35.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:35.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015733+0000", 2024-04-24T14:34:35.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021355699999999999 2024-04-24T14:34:35.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:35.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015773+0000", 2024-04-24T14:34:35.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9554000000000001e-05 2024-04-24T14:34:35.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015817+0000", 2024-04-24T14:34:35.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4325e-05 2024-04-24T14:34:35.586 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.586 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.586 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.587 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:131 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:35.587 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:35.587 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.4826595059999992, 2024-04-24T14:34:35.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014251976, 2024-04-24T14:34:35.587 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.587 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:35.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:35.587 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131 2024-04-24T14:34:35.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006034+0000", 2024-04-24T14:34:35.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006042+0000", 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2970000000000004e-06 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006061+0000", 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8505999999999999e-05 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006071+0000", 2024-04-24T14:34:35.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0409e-05 2024-04-24T14:34:35.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007899+0000", 2024-04-24T14:34:35.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018274789999999999 2024-04-24T14:34:35.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008281+0000", 2024-04-24T14:34:35.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038244700000000001 2024-04-24T14:34:35.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008528+0000", 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024723900000000001 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019070+0000", 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010541867999999999 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020099+0000", 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00102852 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:35.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020226+0000", 2024-04-24T14:34:35.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012744300000000001 2024-04-24T14:34:35.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020287+0000", 2024-04-24T14:34:35.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1253000000000005e-05 2024-04-24T14:34:35.592 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.592 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.592 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.592 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:132 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:35.592 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:35.593 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.4826056530000002, 2024-04-24T14:34:35.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015917981000000001, 2024-04-24T14:34:35.593 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.593 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:35.593 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.593 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:35.593 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:35.593 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 132 2024-04-24T14:34:35.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.593 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:35.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:35.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:35.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:35.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006093+0000", 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3819999999999997e-06 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006103+0000", 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0352e-05 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006110+0000", 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3910000000000003e-06 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015586+0000", 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094759119999999995 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015730+0000", 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014487199999999999 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015886+0000", 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015573799999999999 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020306+0000", 2024-04-24T14:34:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044199399999999998 2024-04-24T14:34:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021887+0000", 2024-04-24T14:34:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015809400000000001 2024-04-24T14:34:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021942+0000", 2024-04-24T14:34:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5071000000000002e-05 2024-04-24T14:34:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.022007+0000", 2024-04-24T14:34:35.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5110000000000005e-05 2024-04-24T14:34:35.598 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.598 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.598 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.598 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.5324.0:127 3.0 e87/69 ondisk, result = 0)", 2024-04-24T14:34:35.598 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:35.598 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.480896585, 2024-04-24T14:34:35.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011245125, 2024-04-24T14:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:35.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:35.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007804+0000", 2024-04-24T14:34:35.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6610000000000003e-06 2024-04-24T14:34:35.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007817+0000", 2024-04-24T14:34:35.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3030999999999999e-05 2024-04-24T14:34:35.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007832+0000", 2024-04-24T14:34:35.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.47e-05 2024-04-24T14:34:35.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018843+0000", 2024-04-24T14:34:35.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011010647 2024-04-24T14:34:35.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018853+0000", 2024-04-24T14:34:35.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319000000000001e-05 2024-04-24T14:34:35.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019043+0000", 2024-04-24T14:34:35.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019055600000000001 2024-04-24T14:34:35.602 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.602 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.602 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.602 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.602 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.602 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:35.604 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:35.606+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:35.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:35.704 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:35.709+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:35.752 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:35.753 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:35.753 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:35.753 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:35.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:35.806 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:35.810+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:35.855 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:35.856 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:35.856 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:35.856 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:35.856 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:35.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:35.906 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:35.909+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:35.971 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:35.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.218562087999999, 2024-04-24T14:34:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:34:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:34:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:34:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:34:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:34:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:34:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:34:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:34:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:34:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:34:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:34:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:34:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:34:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:34:35.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:34:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:34:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:34:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:34:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:34:35.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.218511253999999, 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:34:35.977 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:34:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:34:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:34:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:34:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:34:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:34:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:34:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:34:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:34:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:34:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:34:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:34:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:34:35.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:34:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:34:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:34:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:34:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:34:35.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:34:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.218474946999997, 2024-04-24T14:34:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:34:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:35.983 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:34:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:34:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:34:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:34:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:34:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:34:35.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:34:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:34:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:34:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:34:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:35.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:34:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:34:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:34:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:34:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:34:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:34:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.215063696999998, 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:34:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:34:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:34:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:34:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:34:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:34:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:34:35.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:34:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:34:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.21500202, 2024-04-24T14:34:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:34:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:35.994 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:34:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:35.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:34:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:34:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:34:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:34:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:34:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:34:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:34:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:34:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:34:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:35.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.214967444000003, 2024-04-24T14:34:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:34:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:34:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.000 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:36.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:34:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:34:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:34:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:34:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.214938105000002, 2024-04-24T14:34:36.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:34:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:34:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:34:36.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:34:36.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:34:36.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:34:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:34:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:34:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:34:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:34:36.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.214861523000003, 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.011 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:34:36.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:34:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:34:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:34:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:34:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:34:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:34:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:34:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:34:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:34:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:34:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:34:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:34:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:34:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:34:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:34:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:34:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:34:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:34:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:34:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:34:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.016 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.214813978000002, 2024-04-24T14:34:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:34:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:34:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:34:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:34:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:34:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:34:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:34:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:34:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:34:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:34:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:34:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.214773336, 2024-04-24T14:34:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:34:36.022 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:34:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:34:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:34:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:34:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:34:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:34:36.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:34:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:34:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:34:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:34:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:34:36.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:34:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.214756225000002, 2024-04-24T14:34:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:34:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:34:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:34:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:34:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:34:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:34:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:34:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:34:36.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.033 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:103 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.0151901120000009, 2024-04-24T14:34:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032646351999999997, 2024-04-24T14:34:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 103 2024-04-24T14:34:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959674+0000", 2024-04-24T14:34:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959688+0000", 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959722+0000", 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3936000000000002e-05 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959735+0000", 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2926e-05 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959921+0000", 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185601 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960324+0000", 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040356700000000002 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960839+0000", 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051410599999999996 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.975634+0000", 2024-04-24T14:34:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014795770999999999 2024-04-24T14:34:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992113+0000", 2024-04-24T14:34:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016478982999999999 2024-04-24T14:34:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992228+0000", 2024-04-24T14:34:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114295 2024-04-24T14:34:36.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992323+0000", 2024-04-24T14:34:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5344000000000001e-05 2024-04-24T14:34:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:105 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.0149403340000003, 2024-04-24T14:34:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032581757000000003, 2024-04-24T14:34:36.039 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959925+0000", 2024-04-24T14:34:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959937+0000", 2024-04-24T14:34:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2391999999999999e-05 2024-04-24T14:34:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:34:36.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2535999999999999e-05 2024-04-24T14:34:36.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959970+0000", 2024-04-24T14:34:36.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1039999999999999e-05 2024-04-24T14:34:36.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969531+0000", 2024-04-24T14:34:36.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095603949999999993 2024-04-24T14:34:36.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969669+0000", 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013806499999999999 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970010+0000", 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034062200000000001 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976983+0000", 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069735600000000002 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992413+0000", 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015429841999999999 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992457+0000", 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4169999999999999e-05 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992508+0000", 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0995000000000002e-05 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.045 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:113 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:36.045 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:36.045 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.9947140430000001, 2024-04-24T14:34:36.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014793863000000001, 2024-04-24T14:34:36.045 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.045 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:36.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:36.045 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 113 2024-04-24T14:34:36.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.045 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:36.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:36.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980152+0000", 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980156+0000", 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.619e-06 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980163+0000", 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7070000000000003e-06 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980173+0000", 2024-04-24T14:34:36.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0957e-05 2024-04-24T14:34:36.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983381+0000", 2024-04-24T14:34:36.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032079930000000001 2024-04-24T14:34:36.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983628+0000", 2024-04-24T14:34:36.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024665700000000001 2024-04-24T14:34:36.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983964+0000", 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000336332 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992694+0000", 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087297009999999994 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994878+0000", 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021841439999999998 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994917+0000", 2024-04-24T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8544999999999998e-05 2024-04-24T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994947+0000", 2024-04-24T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9705000000000001e-05 2024-04-24T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:130 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.9690206079999992, 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014646289999999999, 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 130 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005844+0000", 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005853+0000", 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1319999999999994e-06 2024-04-24T14:34:36.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005870+0000", 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7448999999999998e-05 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005876+0000", 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3319999999999997e-06 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005913+0000", 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6835999999999998e-05 2024-04-24T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006017+0000", 2024-04-24T14:34:36.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010358500000000001 2024-04-24T14:34:36.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:36.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006139+0000", 2024-04-24T14:34:36.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001227 2024-04-24T14:34:36.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:36.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.010194+0000", 2024-04-24T14:34:36.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040549380000000001 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020365+0000", 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010170983 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020428+0000", 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2457000000000002e-05 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020492+0000", 2024-04-24T14:34:36.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4592999999999996e-05 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:155 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1601536~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.9353598210000005, 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015144391, 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:36.056 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 155 2024-04-24T14:34:36.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.057 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:36.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:36.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039506+0000", 2024-04-24T14:34:36.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:36.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039766+0000", 2024-04-24T14:34:36.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025994400000000002 2024-04-24T14:34:36.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039780+0000", 2024-04-24T14:34:36.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3366e-05 2024-04-24T14:34:36.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039787+0000", 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4359999999999996e-06 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.050886+0000", 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011098993 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051026+0000", 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001395 2024-04-24T14:34:36.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051211+0000", 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018547099999999999 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054490+0000", 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032785499999999999 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054505+0000", 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5906999999999999e-05 2024-04-24T14:34:36.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:36.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054594+0000", 2024-04-24T14:34:36.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8817e-05 2024-04-24T14:34:36.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054651+0000", 2024-04-24T14:34:36.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7040999999999998e-05 2024-04-24T14:34:36.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.061 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.062 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:156 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1732608~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:36.062 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:36.062 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.9350691149999992, 2024-04-24T14:34:36.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016895194999999998, 2024-04-24T14:34:36.062 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.062 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:36.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:36.062 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 156 2024-04-24T14:34:36.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.062 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:36.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:36.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039797+0000", 2024-04-24T14:34:36.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040065+0000", 2024-04-24T14:34:36.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000268361 2024-04-24T14:34:36.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040086+0000", 2024-04-24T14:34:36.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1024000000000001e-05 2024-04-24T14:34:36.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040094+0000", 2024-04-24T14:34:36.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9379999999999992e-06 2024-04-24T14:34:36.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051936+0000", 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011842518 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052062+0000", 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125482 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052206+0000", 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143953 2024-04-24T14:34:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054715+0000", 2024-04-24T14:34:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025088240000000002 2024-04-24T14:34:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056534+0000", 2024-04-24T14:34:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001819436 2024-04-24T14:34:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056610+0000", 2024-04-24T14:34:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5383000000000007e-05 2024-04-24T14:34:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056693+0000", 2024-04-24T14:34:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3283999999999997e-05 2024-04-24T14:34:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:157 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1863680~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.9347617360000005, 2024-04-24T14:34:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016763674999999999, 2024-04-24T14:34:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157 2024-04-24T14:34:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:36.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040104+0000", 2024-04-24T14:34:36.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040244+0000", 2024-04-24T14:34:36.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014035499999999999 2024-04-24T14:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040255+0000", 2024-04-24T14:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0135999999999999e-05 2024-04-24T14:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040260+0000", 2024-04-24T14:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7919999999999998e-06 2024-04-24T14:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053406+0000", 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013146023999999999 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053531+0000", 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012471800000000001 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053715+0000", 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018395099999999999 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:36.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056711+0000", 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029956520000000001 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056783+0000", 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2010999999999999e-05 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056831+0000", 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8189999999999998e-05 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056869+0000", 2024-04-24T14:34:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7812000000000003e-05 2024-04-24T14:34:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:158 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1994752~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.9345998089999998, 2024-04-24T14:34:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016845334999999999, 2024-04-24T14:34:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2024-04-24T14:34:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040266+0000", 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040379+0000", 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112728 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040387+0000", 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8499999999999994e-06 2024-04-24T14:34:36.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040391+0000", 2024-04-24T14:34:36.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2719999999999999e-06 2024-04-24T14:34:36.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054803+0000", 2024-04-24T14:34:36.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014411548 2024-04-24T14:34:36.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054982+0000", 2024-04-24T14:34:36.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017941699999999999 2024-04-24T14:34:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.055167+0000", 2024-04-24T14:34:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018542100000000001 2024-04-24T14:34:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056718+0000", 2024-04-24T14:34:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015505670000000001 2024-04-24T14:34:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057020+0000", 2024-04-24T14:34:36.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030227500000000002 2024-04-24T14:34:36.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:36.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057070+0000", 2024-04-24T14:34:36.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9611000000000003e-05 2024-04-24T14:34:36.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057112+0000", 2024-04-24T14:34:36.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2373999999999998e-05 2024-04-24T14:34:36.078 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.078 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.078 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:159 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.9291648210000005, 2024-04-24T14:34:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015930650000000001, 2024-04-24T14:34:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2024-04-24T14:34:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045701+0000", 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045707+0000", 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.562e-06 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045728+0000", 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1109999999999999e-05 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045738+0000", 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0016e-05 2024-04-24T14:34:36.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056183+0000", 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010444836000000001 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056470+0000", 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028696899999999998 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057160+0000", 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069004699999999995 2024-04-24T14:34:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057291+0000", 2024-04-24T14:34:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013096200000000001 2024-04-24T14:34:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.061633+0000", 2024-04-24T14:34:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004341414 2024-04-24T14:34:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.084 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:36.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:36.085 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:36.086+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:36.132 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-24T14:34:36.186 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:36.189+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:36.198 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:36.198 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:36.199 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:36.199 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:36.207 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:36.286 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:36.290+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:36.305 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:36.306 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:36.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:36.387 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:36.390+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:36.466 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:36.466 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:36.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.998717577000001, 2024-04-24T14:34:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:34:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:34:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:34:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:34:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:34:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:34:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:34:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:34:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:34:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:34:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.623360873999999, 2024-04-24T14:34:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:34:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:34:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:34:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:34:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:34:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:34:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:34:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:34:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:34:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:34:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:34:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:34:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:34:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:34:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.476 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:36.476 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.621932719999997, 2024-04-24T14:34:36.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:34:36.476 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.476 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.476 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.476 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.476 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.476 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:34:36.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.476 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:34:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:34:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:34:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:34:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:34:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:34:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:34:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:34:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.621905630999997, 2024-04-24T14:34:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:34:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:34:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:34:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:34:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:34:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:34:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:34:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:34:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:34:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:34:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:34:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.621881664, 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:34:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:34:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:34:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:34:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:34:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:34:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:34:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:34:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:34:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:34:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:34:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:34:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:34:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:34:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.607897684999998, 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:34:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:34:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:34:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:34:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:34:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:34:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:34:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:34:36.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:34:36.494 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.494 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.494 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.494 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.494 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:36.494 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.607781373999998, 2024-04-24T14:34:36.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:34:36.494 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.494 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.495 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:34:36.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.495 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:36.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:36.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:34:36.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:36.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:34:36.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:34:36.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:34:36.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:34:36.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:34:36.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:34:36.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:34:36.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:34:36.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:34:36.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:34:36.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:34:36.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:34:36.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:34:36.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:34:36.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.500 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.500 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.500 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:36.500 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.607733715999998, 2024-04-24T14:34:36.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:34:36.500 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.500 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.501 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:34:36.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.501 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:36.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:36.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:34:36.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:34:36.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:34:36.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:34:36.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:34:36.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:34:36.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:34:36.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:34:36.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:34:36.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:34:36.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:34:36.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.505 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.506 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:36.506 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.603696417999998, 2024-04-24T14:34:36.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:34:36.506 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.506 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.506 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.506 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.506 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.506 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:34:36.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.506 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:34:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:34:36.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:34:36.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:34:36.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:34:36.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:34:36.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:34:36.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:36.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:34:36.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:34:36.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:34:36.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:34:36.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:34:36.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:34:36.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:34:36.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:34:36.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:34:36.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.603632658999999, 2024-04-24T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.603607169, 2024-04-24T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.603583419000003, 2024-04-24T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:34:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:34:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:34:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:34:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:34:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:34:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:34:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:34:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:34:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:34:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:34:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.603544906000003, 2024-04-24T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:34:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:34:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:34:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:34:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:34:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:34:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:34:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:34:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:34:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.603516708999997, 2024-04-24T14:34:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:34:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:34:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:34:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:34:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:34:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:34:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:34:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:34:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:34:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:34:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:34:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:34:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:34:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:34:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:34:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:34:36.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:34:36.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:34:36.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:34:36.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:34:36.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:34:36.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.603495252000002, 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.540 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:34:36.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.603476409999999, 2024-04-24T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:34:36.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:34:36.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:34:36.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:34:36.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:34:36.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:34:36.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:34:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:34:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:34:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:34:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:34:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:34:36.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:34:36.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:34:36.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:34:36.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:34:36.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:34:36.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:34:36.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:34:36.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:34:36.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.555 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.556 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:36.556 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:36.556 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.390644260999998, 2024-04-24T14:34:36.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:34:36.556 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:36.556 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.556 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:34:36.556 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:36.556 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:34:36.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.556 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:36.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:36.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:34:36.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:36.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:34:36.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:34:36.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:34:36.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:36.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:34:36.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:34:36.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:34:36.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:34:36.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:34:36.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:34:36.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:34:36.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:34:36.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:34:36.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:34:36.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:34:36.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:34:36.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.561 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:39 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1753088~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:36.562 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:36.562 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.450871845, 2024-04-24T14:34:36.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01139394, 2024-04-24T14:34:36.562 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.562 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:36.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:36.562 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 39 2024-04-24T14:34:36.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.562 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018967+0000", 2024-04-24T14:34:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019242+0000", 2024-04-24T14:34:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027504500000000002 2024-04-24T14:34:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019274+0000", 2024-04-24T14:34:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2472999999999999e-05 2024-04-24T14:34:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019288+0000", 2024-04-24T14:34:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3283e-05 2024-04-24T14:34:36.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026753+0000", 2024-04-24T14:34:36.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074651120000000003 2024-04-24T14:34:36.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026971+0000", 2024-04-24T14:34:36.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021810800000000001 2024-04-24T14:34:36.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:36.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.027211+0000", 2024-04-24T14:34:36.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023975 2024-04-24T14:34:36.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:36.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030200+0000", 2024-04-24T14:34:36.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029890709999999998 2024-04-24T14:34:36.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030252+0000", 2024-04-24T14:34:36.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1926999999999999e-05 2024-04-24T14:34:36.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:36.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030312+0000", 2024-04-24T14:34:36.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0602e-05 2024-04-24T14:34:36.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030363+0000", 2024-04-24T14:34:36.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0342e-05 2024-04-24T14:34:36.567 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.567 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.567 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.567 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:40 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1884160~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:36.568 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:36.568 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.4505358709999996, 2024-04-24T14:34:36.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011199064, 2024-04-24T14:34:36.568 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.568 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:36.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:36.568 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 40 2024-04-24T14:34:36.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.568 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:36.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:36.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019303+0000", 2024-04-24T14:34:36.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:36.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019729+0000", 2024-04-24T14:34:36.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042620899999999998 2024-04-24T14:34:36.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019769+0000", 2024-04-24T14:34:36.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9847e-05 2024-04-24T14:34:36.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019797+0000", 2024-04-24T14:34:36.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8347000000000002e-05 2024-04-24T14:34:36.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028299+0000", 2024-04-24T14:34:36.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085010629999999997 2024-04-24T14:34:36.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028385+0000", 2024-04-24T14:34:36.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6824999999999996e-05 2024-04-24T14:34:36.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:36.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028510+0000", 2024-04-24T14:34:36.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012483000000000001 2024-04-24T14:34:36.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030410+0000", 2024-04-24T14:34:36.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001899612 2024-04-24T14:34:36.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:36.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030416+0000", 2024-04-24T14:34:36.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7699999999999998e-06 2024-04-24T14:34:36.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:36.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030461+0000", 2024-04-24T14:34:36.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5157999999999997e-05 2024-04-24T14:34:36.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030504+0000", 2024-04-24T14:34:36.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2988999999999998e-05 2024-04-24T14:34:36.573 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.573 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.573 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.573 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:41 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 2015232~81920] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:36.574 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:36.574 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.4500271639999998, 2024-04-24T14:34:36.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012345257, 2024-04-24T14:34:36.574 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:36.574 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.574 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:36.574 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:36.574 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:36.574 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 41 2024-04-24T14:34:36.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.574 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:36.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:36.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:36.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:36.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:36.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:36.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:36.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019812+0000", 2024-04-24T14:34:36.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:36.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:36.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019975+0000", 2024-04-24T14:34:36.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016325700000000001 2024-04-24T14:34:36.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:36.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020025+0000", 2024-04-24T14:34:36.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9484999999999998e-05 2024-04-24T14:34:36.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020035+0000", 2024-04-24T14:34:36.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0444e-05 2024-04-24T14:34:36.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:36.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029150+0000", 2024-04-24T14:34:36.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091150160000000001 2024-04-24T14:34:36.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:36.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029352+0000", 2024-04-24T14:34:36.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020166499999999999 2024-04-24T14:34:36.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:36.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029611+0000", 2024-04-24T14:34:36.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025949600000000001 2024-04-24T14:34:36.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:36.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030518+0000", 2024-04-24T14:34:36.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090633300000000005 2024-04-24T14:34:36.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032088+0000", 2024-04-24T14:34:36.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015703290000000001 2024-04-24T14:34:36.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:36.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032130+0000", 2024-04-24T14:34:36.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2273e-05 2024-04-24T14:34:36.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:36.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:36.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:36.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032159+0000", 2024-04-24T14:34:36.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8387000000000001e-05 2024-04-24T14:34:36.579 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.579 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.579 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.579 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:36.579 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:36.579 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:36.581 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:36.582 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:36.585+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:36.682 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:36.686+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:36.683 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:36.683 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:36.683 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:36.683 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:36.693 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:36.784 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:36.789+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:36.794 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:36.794 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:36.794 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:36.795 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:36.795 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:36.804 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:36.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:34:36.884 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:36.886+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:36.906 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:36.906 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:36.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:36.906 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:36.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.906 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:36.906 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:36.906 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 134.282647965, 2024-04-24T14:34:36.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:36.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:34:36.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:36.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:34:36.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:34:36.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:34:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:34:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:34:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:34:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:34:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:34:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:36.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:34:36.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:34:36.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:36.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:34:36.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:34:36.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:34:36.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:34:36.912 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.912 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.912 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.912 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:36.912 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:36.912 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 134.282470878, 2024-04-24T14:34:36.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:34:36.912 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.912 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.912 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.913 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:36.913 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:36.913 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:34:36.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.913 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:36.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:36.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:34:36.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:36.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:34:36.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:34:36.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:34:36.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:34:36.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:34:36.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:34:36.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:34:36.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:34:36.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:36.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:34:36.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:34:36.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:36.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:34:36.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:34:36.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:34:36.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:34:36.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:36.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:34:36.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:34:36.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:36.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:34:36.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:34:36.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:34:36.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:34:36.918 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.918 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.918 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.918 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:36.918 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:36.918 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 134.282366964, 2024-04-24T14:34:36.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:34:36.918 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.918 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.918 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.918 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:36.919 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:36.919 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:34:36.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.919 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:36.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:36.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:34:36.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:36.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:34:36.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:34:36.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:36.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:34:36.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:34:36.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:36.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:34:36.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:34:36.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:34:36.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:34:36.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:36.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:34:36.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:34:36.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:36.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:34:36.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:34:36.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:34:36.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:34:36.923 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.923 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.924 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.924 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:36.924 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:36.924 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 134.28197851300001, 2024-04-24T14:34:36.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:34:36.924 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.924 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.924 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.924 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:36.924 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:36.924 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:34:36.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.925 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:36.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:36.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:34:36.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:36.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:34:36.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:34:36.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:34:36.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:34:36.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:34:36.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:34:36.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:34:36.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:34:36.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:36.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:34:36.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:34:36.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:36.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:34:36.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:34:36.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:34:36.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:34:36.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:36.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:34:36.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.930 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:36.930 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:36.930 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 134.281893301, 2024-04-24T14:34:36.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:34:36.930 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.930 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.930 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.930 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:36.930 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:36.930 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:36.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.930 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:36.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:36.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:34:36.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:36.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:34:36.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:34:36.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:34:36.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:34:36.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:34:36.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:34:36.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:34:36.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:34:36.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:36.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:34:36.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:34:36.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:36.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:34:36.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:34:36.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:34:36.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:34:36.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:36.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:34:36.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:34:36.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:36.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:34:36.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:34:36.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:34:36.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:34:36.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.935 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.936 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:36.936 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:36.936 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 134.28165641999999, 2024-04-24T14:34:36.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:34:36.936 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.936 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.936 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.936 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:36.936 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:36.936 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:34:36.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.936 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:36.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:36.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:34:36.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:36.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:34:36.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:34:36.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:34:36.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:34:36.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:36.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:34:36.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:34:36.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:34:36.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:34:36.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:36.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:34:36.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:34:36.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:36.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:34:36.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:34:36.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:34:36.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:34:36.941 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.941 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.941 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.942 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:36.942 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:36.942 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 134.281574898, 2024-04-24T14:34:36.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:34:36.942 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.942 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.942 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.942 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:36.942 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:36.942 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:34:36.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.942 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:36.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:36.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:34:36.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:34:36.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:34:36.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:34:36.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:34:36.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:34:36.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:36.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:34:36.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:34:36.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:36.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:34:36.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:34:36.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:34:36.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:34:36.947 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.947 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.947 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.947 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:36.947 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:36.947 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 133.84966255099999, 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:36.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:34:36.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:34:36.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:34:36.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:34:36.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:34:36.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:34:36.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:34:36.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:34:36.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:34:36.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:36.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:34:36.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:34:36.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:36.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:34:36.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:34:36.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:34:36.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:34:36.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:36.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:34:36.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:34:36.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:36.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:34:36.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:34:36.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:34:36.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:34:36.956 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.956 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.956 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.956 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:36.956 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:36.956 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 133.849634251, 2024-04-24T14:34:36.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:34:36.957 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.957 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.957 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.957 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:36.957 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:36.957 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:34:36.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.957 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:36.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:34:36.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:34:36.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:34:36.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:34:36.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:34:36.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:34:36.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:34:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:34:36.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:34:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 133.84954654699999, 2024-04-24T14:34:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:34:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.962 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:34:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:34:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:34:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:34:36.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:34:36.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:34:36.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:34:36.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:34:36.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:34:36.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:36.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:34:36.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:34:36.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:36.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:34:36.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:34:36.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:34:36.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:34:36.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:36.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:34:36.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:34:36.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:36.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:34:36.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:34:36.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:34:36.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:34:36.967 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.968 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.968 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.968 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:36.968 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:36.968 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 133.84942374600001, 2024-04-24T14:34:36.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:34:36.968 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.968 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.968 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.968 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:36.968 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:36.969 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:34:36.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.969 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:36.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:36.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:34:36.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:34:36.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:34:36.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:34:36.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:34:36.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:34:36.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:34:36.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:34:36.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:34:36.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:36.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:34:36.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:34:36.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:36.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:34:36.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:34:36.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.973 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:34:36.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:34:36.973 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.974 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.974 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.974 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:34:36.974 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:36.974 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.543179186000003, 2024-04-24T14:34:36.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:34:36.974 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.974 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:36.974 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.974 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:34:36.974 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:34:36.975 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:34:36.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.975 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:36.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:36.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:34:36.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:34:36.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:34:36.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:34:36.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:34:36.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:34:36.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:34:36.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:34:36.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:34:36.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:34:36.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:36.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:34:36.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:34:36.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:34:36.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:34:36.979 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.979 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.979 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.979 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:36.979 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:36.979 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 56.849055186999998, 2024-04-24T14:34:36.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:34:36.979 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.979 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:36.980 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.980 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:34:36.980 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:34:36.980 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:34:36.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.980 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:36.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:34:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:34:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:34:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.982 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.982 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:34:36.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:34:36.982 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.982 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.982 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:34:36.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:34:36.982 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.982 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.982 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:34:36.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:34:36.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:34:36.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:34:36.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:34:36.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:34:36.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:34:36.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:34:36.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:34:36.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:34:36.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:36.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:34:36.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:34:36.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:34:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:34:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.889695220999997, 2024-04-24T14:34:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:34:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.985 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:36.986 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.986 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:36.986 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:36.986 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:34:36.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.986 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:36.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:34:36.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:34:36.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:34:36.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:34:36.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:34:36.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:34:36.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:34:36.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:34:36.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:34:36.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:34:36.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:34:36.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:34:36.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:34:36.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:34:36.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:36.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:34:36.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:34:36.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:34:36.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:34:36.991 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.991 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.991 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.991 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:36.991 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:36.991 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.889614666, 2024-04-24T14:34:36.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:34:36.991 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.991 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:36.991 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.991 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:36.992 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:36.992 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:34:36.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.992 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:36.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:36.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:34:36.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:34:36.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:34:36.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:34:36.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:34:36.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:36.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:34:36.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:34:36.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:34:36.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:34:36.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:36.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:34:36.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:34:36.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:34:36.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:34:36.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:36.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:34:36.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:34:36.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.996 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:36.997 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:36.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.997 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:36.997 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:36.997 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.889566936999998, 2024-04-24T14:34:36.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:34:36.997 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:36.997 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:36.997 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:36.997 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:36.997 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:36.997 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:34:36.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.998 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:36.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:36.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:36.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:36.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:36.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:36.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:36.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:34:36.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:36.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:36.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:34:36.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:34:36.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:36.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:36.999 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:34:36.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:34:36.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:36.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:34:37.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:34:37.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:37.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:34:37.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:34:37.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:34:37.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:34:37.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:37.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:34:37.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:34:37.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:37.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:34:37.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:34:37.002 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:37.002 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:37.002 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:37.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.003 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:37.003 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:37.003 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.889539173999999, 2024-04-24T14:34:37.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:34:37.003 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:37.003 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:37.003 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:37.003 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:37.003 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:37.003 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:34:37.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.003 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:37.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:37.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:37.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:37.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:37.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:37.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:37.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:37.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:34:37.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:37.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:37.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:34:37.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:34:37.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:37.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:34:37.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:34:37.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:37.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:34:37.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:34:37.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:34:37.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:34:37.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:37.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:34:37.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:34:37.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:34:37.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:34:37.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:37.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:34:37.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:34:37.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.007 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.007 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:34:37.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:34:37.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:37.008 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:37.009 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.889499739000001, 2024-04-24T14:34:37.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:34:37.009 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:37.009 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:37.009 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:37.009 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:37.009 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:37.009 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:34:37.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.009 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:37.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:37.009 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:37.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:37.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:37.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:37.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:37.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:37.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:34:37.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:37.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:34:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:34:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:34:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:34:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:34:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:34:37.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:34:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:34:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:34:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:34:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:34:37.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:34:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:34:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:34:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:34:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:34:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:37.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:34:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:34:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:34:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:34:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:37.014 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.889474755000002, 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:37.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:34:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:34:37.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:34:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:34:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:34:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:34:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:34:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:34:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:34:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:34:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:34:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:34:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:34:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:34:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:34:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:34:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:34:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:34:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:34:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:34:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:34:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 37.889449096, 2024-04-24T14:34:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:34:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:37.020 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:34:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:37.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:34:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:34:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:34:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:34:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:34:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:34:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:34:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:34:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:34:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:34:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:34:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:34:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:34:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:34:37.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:34:37.025 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:37.026 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:37.026 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:37.027 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:37.027 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:37.027 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:37.028 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:37.028 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:37.030+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:37.028 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:37.038 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:37.059 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:37.062 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:37.062 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:37.062 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:37.062 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:37.062 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:37.062 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:37.128 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:37.133+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:37.135 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:37.135 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:37.144 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:37.229 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:37.234+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:37.289 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:37.290 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:37.298 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:37.330 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:37.334+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:37.398 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:37.398 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:37.407 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:37.431 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:37.434+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:37.508 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:37.508 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:37.517 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:37.531 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:37.534+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:37.620 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:37.620 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:37.630 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:37.631 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:37.637+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:37.732 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:37.732 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:37.732 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:37.737+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:37.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:37.834 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:37.837+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:37.849 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:37.849 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:37.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:37.935 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:37.941+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:37.962 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:37.962 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:37.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:38.036 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:38.041+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:38.077 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:38.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:38.137 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:38.141+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:38.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T14:34:38.202 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:38.202 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:38.212 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:38.238 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:38.242+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:38.311 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:38.311 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:38.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:38.340 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:38.345+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:38.411 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:38.411 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:38.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:38.466 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:38.470+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:38.518 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:38.518 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:38.527 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:38.542 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:38.545+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:38.626 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:38.627 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:38.636 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:38.642 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:38.646+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:38.734 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:38.734 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:38.743 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:38.749+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:38.744 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:38.844 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:38.846+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:38.850 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:38.850 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:38.860 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:38.945 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:38.946+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:38.956 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:38.956 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:38.965 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:39.046 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:39.050+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:39.064 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:39.065 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:39.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:34:39.075 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:39.147 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:39.150+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:39.172 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-24T14:34:39.179 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:39.179 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:39.189 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:39.211 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:39.232 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:39.248 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:39.250+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:39.252 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:39.274 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:39.284 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:39.284 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:39.294 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:39.297 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:39.320 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:39.323 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:39.323 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:39.323 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:39.323 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:39.323 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:39.323 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:39.348 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:39.353+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:39.392 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 57.731280718999997, 2024-04-24T14:34:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:34:39.392 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:34:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:34:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:34:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:34:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:34:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:34:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:34:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:34:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:34:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:34:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:34:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:34:39.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:34:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 57.728246192999997, 2024-04-24T14:34:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:34:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.397 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:34:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:34:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:34:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:34:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:34:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:34:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:34:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 57.728222006999999, 2024-04-24T14:34:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:34:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:34:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:39.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:34:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:34:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:34:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:34:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:34:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:34:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:34:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:34:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:34:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:34:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:34:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:34:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 57.728204623000003, 2024-04-24T14:34:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:34:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.406 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:34:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:34:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:34:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:34:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:34:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:34:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:34:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:34:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:34:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:34:39.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:34:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:34:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:34:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:34:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 57.713768530999999, 2024-04-24T14:34:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:34:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:34:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.411 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:34:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:34:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:34:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:34:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:34:39.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:34:39.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:34:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:34:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 57.713736232000002, 2024-04-24T14:34:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:34:39.415 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:34:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:34:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:34:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:34:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:34:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:34:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:34:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:34:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:34:39.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.419 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 57.699483954999998, 2024-04-24T14:34:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:34:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:34:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:34:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:34:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:34:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:34:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:34:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:34:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:34:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:34:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:34:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:34:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:34:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:34:39.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:34:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 57.699454854999999, 2024-04-24T14:34:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:34:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:39.424 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:34:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:39.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:39.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:34:39.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:34:39.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:34:39.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:34:39.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:34:39.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:34:39.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:34:39.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:34:39.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:34:39.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:34:39.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:34:39.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:34:39.428 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.428 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.428 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.428 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:39.428 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:39.428 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 57.699428924999999, 2024-04-24T14:34:39.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:34:39.429 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.429 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:39.429 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.429 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:39.429 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:39.429 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:34:39.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.429 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:39.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:39.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:39.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:39.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:34:39.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:34:39.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:34:39.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:34:39.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:34:39.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:34:39.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:34:39.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:34:39.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:34:39.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:34:39.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:34:39.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:34:39.432 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 57.683708461000002, 2024-04-24T14:34:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:34:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:39.433 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:34:39.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.434 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:39.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:39.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:34:39.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:39.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:34:39.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:34:39.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:34:39.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:34:39.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:34:39.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:34:39.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:34:39.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:34:39.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:34:39.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:34:39.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:34:39.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:34:39.437 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.437 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.437 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.437 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:39.437 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:39.437 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 57.683676529000003, 2024-04-24T14:34:39.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:34:39.438 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.438 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:39.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:39.438 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:39.438 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:34:39.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.438 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:39.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:39.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:34:39.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:39.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:34:39.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:34:39.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:34:39.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:34:39.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:34:39.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:34:39.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:34:39.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:34:39.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:34:39.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:34:39.441 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.441 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.441 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.441 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:39.441 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:39.442 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 57.683609013000002, 2024-04-24T14:34:39.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:34:39.442 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.442 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:39.442 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.442 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:39.442 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:39.442 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:34:39.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.442 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:39.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:39.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:34:39.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:39.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:34:39.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:34:39.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:34:39.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:34:39.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:34:39.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:34:39.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:34:39.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:34:39.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:34:39.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:34:39.445 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.445 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.445 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.446 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:34:39.446 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:39.446 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 57.626196129999997, 2024-04-24T14:34:39.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:34:39.446 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.446 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.446 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:39.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:39.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:39.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:39.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:34:39.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:34:39.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:34:39.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:34:39.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:34:39.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:34:39.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:34:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:34:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:34:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:34:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:34:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:34:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:34:39.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:34:39.450 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.450 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.450 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.450 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:103 3.1 e87/74 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head v 87'261, mlcod=87'258)", 2024-04-24T14:34:39.450 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:39.450 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.434914925999999, 2024-04-24T14:34:39.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019745728000000001, 2024-04-24T14:34:39.450 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.450 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.451 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:39.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:39.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960679+0000", 2024-04-24T14:34:39.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:39.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960692+0000", 2024-04-24T14:34:39.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3709999999999999e-05 2024-04-24T14:34:39.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960722+0000", 2024-04-24T14:34:39.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9791999999999999e-05 2024-04-24T14:34:39.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960733+0000", 2024-04-24T14:34:39.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.128e-05 2024-04-24T14:34:39.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960857+0000", 2024-04-24T14:34:39.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000123981 2024-04-24T14:34:39.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960926+0000", 2024-04-24T14:34:39.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8881999999999995e-05 2024-04-24T14:34:39.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980353+0000", 2024-04-24T14:34:39.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019426853000000001 2024-04-24T14:34:39.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980426+0000", 2024-04-24T14:34:39.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2803000000000001e-05 2024-04-24T14:34:39.454 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.454 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.454 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.454 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:105 3.1 e87/74 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head v 87'263, mlcod=87'258)", 2024-04-24T14:34:39.455 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:39.455 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.425788391999999, 2024-04-24T14:34:39.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014864743999999999, 2024-04-24T14:34:39.455 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.455 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.455 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:39.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:39.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969805+0000", 2024-04-24T14:34:39.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:39.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969820+0000", 2024-04-24T14:34:39.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5116999999999999e-05 2024-04-24T14:34:39.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969852+0000", 2024-04-24T14:34:39.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2135999999999999e-05 2024-04-24T14:34:39.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969864+0000", 2024-04-24T14:34:39.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.189e-05 2024-04-24T14:34:39.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972122+0000", 2024-04-24T14:34:39.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022580389999999999 2024-04-24T14:34:39.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972165+0000", 2024-04-24T14:34:39.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2429000000000003e-05 2024-04-24T14:34:39.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984619+0000", 2024-04-24T14:34:39.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012454735 2024-04-24T14:34:39.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984672+0000", 2024-04-24T14:34:39.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2077999999999998e-05 2024-04-24T14:34:39.458 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.459 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.459 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.459 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:112 3.1 e87/74 3:a983798d:::rbd_data.12ff64619e2d.0000000000000006:head v 87'265, mlcod=87'258)", 2024-04-24T14:34:39.459 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:39.459 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.415449556, 2024-04-24T14:34:39.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012821978, 2024-04-24T14:34:39.459 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.459 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.459 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980144+0000", 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980159+0000", 2024-04-24T14:34:39.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4769999999999999e-05 2024-04-24T14:34:39.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980189+0000", 2024-04-24T14:34:39.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0419000000000001e-05 2024-04-24T14:34:39.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980213+0000", 2024-04-24T14:34:39.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4036000000000002e-05 2024-04-24T14:34:39.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980452+0000", 2024-04-24T14:34:39.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00023921500000000001 2024-04-24T14:34:39.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980468+0000", 2024-04-24T14:34:39.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5843999999999999e-05 2024-04-24T14:34:39.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992932+0000", 2024-04-24T14:34:39.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012463314 2024-04-24T14:34:39.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992968+0000", 2024-04-24T14:34:39.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6075999999999998e-05 2024-04-24T14:34:39.463 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.463 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.463 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.463 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:113 3.1 e87/74 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head v 87'267, mlcod=87'258)", 2024-04-24T14:34:39.463 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:39.463 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.411914832000001, 2024-04-24T14:34:39.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010561022999999999, 2024-04-24T14:34:39.463 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.464 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.464 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:39.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:39.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983679+0000", 2024-04-24T14:34:39.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:39.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983699+0000", 2024-04-24T14:34:39.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0392000000000001e-05 2024-04-24T14:34:39.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983737+0000", 2024-04-24T14:34:39.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8340000000000002e-05 2024-04-24T14:34:39.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983756+0000", 2024-04-24T14:34:39.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8402000000000001e-05 2024-04-24T14:34:39.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984706+0000", 2024-04-24T14:34:39.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095024400000000005 2024-04-24T14:34:39.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984741+0000", 2024-04-24T14:34:39.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4721000000000002e-05 2024-04-24T14:34:39.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994211+0000", 2024-04-24T14:34:39.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094708559999999997 2024-04-24T14:34:39.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994241+0000", 2024-04-24T14:34:39.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9941e-05 2024-04-24T14:34:39.467 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.467 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.467 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:115 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.411460795, 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010347242, 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:39.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:39.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984132+0000", 2024-04-24T14:34:39.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:39.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984142+0000", 2024-04-24T14:34:39.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7370000000000001e-06 2024-04-24T14:34:39.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984154+0000", 2024-04-24T14:34:39.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2288e-05 2024-04-24T14:34:39.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984163+0000", 2024-04-24T14:34:39.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3749999999999992e-06 2024-04-24T14:34:39.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984246+0000", 2024-04-24T14:34:39.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2962999999999998e-05 2024-04-24T14:34:39.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984473+0000", 2024-04-24T14:34:39.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00022696399999999999 2024-04-24T14:34:39.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:39.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984771+0000", 2024-04-24T14:34:39.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00029739499999999999 2024-04-24T14:34:39.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:39.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992998+0000", 2024-04-24T14:34:39.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082275750000000009 2024-04-24T14:34:39.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:39.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994299+0000", 2024-04-24T14:34:39.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013005289999999999 2024-04-24T14:34:39.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994403+0000", 2024-04-24T14:34:39.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010376499999999999 2024-04-24T14:34:39.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994482+0000", 2024-04-24T14:34:39.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9013999999999999e-05 2024-04-24T14:34:39.473 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.473 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.473 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.473 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:128 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:39.474 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:39.474 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.390269867000001, 2024-04-24T14:34:39.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011784144, 2024-04-24T14:34:39.474 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.474 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.474 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.474 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:34:39.474 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:39.474 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 128 2024-04-24T14:34:39.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.474 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:39.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:39.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005324+0000", 2024-04-24T14:34:39.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:39.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005332+0000", 2024-04-24T14:34:39.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2989999999999997e-06 2024-04-24T14:34:39.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005354+0000", 2024-04-24T14:34:39.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1705999999999999e-05 2024-04-24T14:34:39.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005362+0000", 2024-04-24T14:34:39.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5599999999999996e-06 2024-04-24T14:34:39.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005418+0000", 2024-04-24T14:34:39.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6685999999999999e-05 2024-04-24T14:34:39.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005535+0000", 2024-04-24T14:34:39.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117008 2024-04-24T14:34:39.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:39.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005732+0000", 2024-04-24T14:34:39.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019718700000000001 2024-04-24T14:34:39.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:39.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016440+0000", 2024-04-24T14:34:39.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010707787999999999 2024-04-24T14:34:39.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:39.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016913+0000", 2024-04-24T14:34:39.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047270400000000001 2024-04-24T14:34:39.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017047+0000", 2024-04-24T14:34:39.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013434699999999999 2024-04-24T14:34:39.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017109+0000", 2024-04-24T14:34:39.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2192000000000004e-05 2024-04-24T14:34:39.479 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.479 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.479 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.479 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:130 3.1 e87/74 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 87'269, mlcod=87'265)", 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.389817107000001, 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013945631999999999, 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:39.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005777+0000", 2024-04-24T14:34:39.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:39.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005791+0000", 2024-04-24T14:34:39.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3743999999999999e-05 2024-04-24T14:34:39.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005803+0000", 2024-04-24T14:34:39.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2330999999999999e-05 2024-04-24T14:34:39.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005810+0000", 2024-04-24T14:34:39.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8990000000000004e-06 2024-04-24T14:34:39.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005888+0000", 2024-04-24T14:34:39.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7872000000000003e-05 2024-04-24T14:34:39.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005911+0000", 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3156e-05 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019642+0000", 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013730456 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019724+0000", 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1981e-05 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.483 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.484 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.484 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.484 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.484 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:39.485 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:39.486 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:39.489+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:39.586 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:39.589+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:39.587 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:39.588 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:39.588 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:39.588 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:39.597 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:39.687 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:39.690+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:39.693 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:39.694 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:39.694 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:39.694 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:39.694 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:39.703 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:39.788 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:39.793+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:39.803 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:39.804 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:39.804 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:39.804 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:39.804 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.804 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:34:39.804 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:39.804 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.73565996, 2024-04-24T14:34:39.804 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:34:39.804 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.804 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.804 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.804 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.804 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.805 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:39.805 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.805 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.805 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.805 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.805 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:39.805 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.805 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.805 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.805 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.805 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:34:39.806 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:39.806 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.806 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.806 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.806 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:34:39.806 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:34:39.806 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.806 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.806 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.806 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:34:39.806 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:34:39.806 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.807 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.807 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.807 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:34:39.807 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:34:39.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.807 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.807 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.807 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:34:39.807 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:34:39.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.807 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.807 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.807 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:34:39.807 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:34:39.808 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.808 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.808 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.808 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:34:39.808 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:34:39.808 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.808 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.808 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.808 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:34:39.808 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:34:39.808 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.809 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.809 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.809 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.809 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.809 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:34:39.809 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:39.809 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.740845381, 2024-04-24T14:34:39.809 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:34:39.809 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.809 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.809 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.809 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.809 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:34:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:39.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.811 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.811 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:34:39.811 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:34:39.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.811 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.811 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:34:39.811 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:34:39.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.812 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:34:39.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:34:39.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.812 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:34:39.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:34:39.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.812 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:34:39.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:34:39.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.813 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.813 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:34:39.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:34:39.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.813 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.813 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:34:39.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:34:39.813 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.813 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.813 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.814 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.814 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.814 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:34:39.814 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:39.814 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.729916123000002, 2024-04-24T14:34:39.814 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:34:39.814 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.814 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.814 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.814 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.814 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.814 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:39.815 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.815 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.815 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.815 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:39.815 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.815 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.815 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.815 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:34:39.815 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:39.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.815 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:34:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:34:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:34:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:34:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:34:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:34:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.817 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.817 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:34:39.817 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:34:39.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.817 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.817 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:34:39.817 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:34:39.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.818 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.818 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:34:39.818 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:34:39.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.818 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.818 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.818 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:34:39.818 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:34:39.818 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.818 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.818 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.819 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.819 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:34:39.819 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:39.819 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.718960041999999, 2024-04-24T14:34:39.819 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:34:39.819 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.819 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.819 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.819 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.819 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.819 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:39.819 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.819 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.820 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.820 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.820 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:39.820 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.820 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.820 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.820 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:34:39.820 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:39.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.821 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.821 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.821 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:34:39.821 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:34:39.821 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.821 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.821 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.821 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:34:39.821 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:34:39.821 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.821 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.821 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.822 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:34:39.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:34:39.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.822 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.822 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:34:39.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:34:39.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.822 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.822 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:34:39.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:34:39.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:34:39.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:34:39.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:34:39.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:34:39.823 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.823 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.824 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.824 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.824 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:34:39.824 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:39.824 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.707390676999999, 2024-04-24T14:34:39.824 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:34:39.824 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.824 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.824 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:39.825 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.825 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.825 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:39.825 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.825 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.825 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:34:39.825 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:39.826 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.826 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.826 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.826 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:34:39.826 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:34:39.826 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.826 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.826 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.826 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:34:39.826 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:34:39.826 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.826 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.827 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:34:39.827 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:34:39.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.827 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:34:39.827 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:34:39.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.827 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:34:39.827 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:34:39.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.828 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.828 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:34:39.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:34:39.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.828 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.828 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:34:39.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:34:39.828 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.828 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.829 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.829 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:34:39.829 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:39.829 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.454271018, 2024-04-24T14:34:39.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:34:39.829 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.829 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.829 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.830 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:39.830 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.830 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.830 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:39.830 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.830 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.830 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:34:39.830 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:39.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.831 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.831 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:34:39.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:34:39.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.831 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.831 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:34:39.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:34:39.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.831 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:34:39.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:34:39.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:34:39.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:34:39.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:34:39.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:34:39.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.833 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.833 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:34:39.833 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:34:39.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.833 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.833 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:34:39.833 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:34:39.833 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.833 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.833 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.834 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:34:39.834 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:39.834 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.44352585, 2024-04-24T14:34:39.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:34:39.834 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.834 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.834 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.834 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.834 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:39.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.835 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:39.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.835 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:34:39.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:39.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:34:39.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:34:39.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:34:39.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:34:39.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:34:39.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:34:39.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:34:39.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:34:39.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:34:39.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:34:39.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:34:39.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:34:39.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:34:39.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:34:39.839 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.839 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.839 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.839 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:34:39.839 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:39.839 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.431159573999999, 2024-04-24T14:34:39.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:34:39.839 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.839 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.839 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:39.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:39.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:34:39.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:39.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:34:39.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:34:39.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:34:39.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:34:39.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:34:39.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:34:39.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:34:39.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:34:39.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:34:39.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:34:39.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:34:39.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:34:39.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:34:39.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:34:39.844 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.844 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.844 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.844 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:34:39.844 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:39.844 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.306223453000001, 2024-04-24T14:34:39.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:34:39.844 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.844 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.844 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:39.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:39.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:34:39.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:39.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:34:39.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:34:39.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:34:39.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:34:39.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:34:39.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:34:39.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:34:39.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:34:39.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:34:39.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:34:39.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:34:39.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:34:39.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:34:39.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:34:39.848 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.849 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.849 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.849 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:34:39.849 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:39.849 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.286835609999997, 2024-04-24T14:34:39.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:34:39.849 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.849 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.849 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:39.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:39.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:34:39.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:39.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:34:39.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:34:39.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:34:39.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:34:39.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:34:39.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:34:39.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:34:39.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:34:39.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:34:39.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:34:39.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:34:39.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:34:39.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:34:39.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:34:39.853 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.854 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.854 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.854 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:34:39.854 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:39.854 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.277107766999997, 2024-04-24T14:34:39.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:34:39.854 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.854 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.855 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:39.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:39.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:34:39.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:39.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:34:39.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:34:39.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:34:39.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:34:39.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:34:39.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:34:39.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:34:39.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:34:39.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:34:39.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:34:39.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:34:39.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:34:39.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:34:39.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:34:39.859 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.859 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.859 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.859 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:34:39.859 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:39.859 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.116998086999999, 2024-04-24T14:34:39.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:34:39.860 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.860 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.860 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:39.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:39.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:34:39.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:39.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:34:39.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:34:39.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:34:39.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:34:39.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:34:39.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:34:39.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:34:39.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:34:39.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:34:39.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:34:39.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:34:39.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:34:39.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:34:39.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:34:39.864 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.864 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.864 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.864 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:34:39.864 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:39.864 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.107269090999999, 2024-04-24T14:34:39.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:34:39.865 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.865 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.865 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:39.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:39.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:34:39.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:39.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:34:39.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:34:39.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:34:39.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:34:39.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:34:39.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:34:39.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:34:39.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:34:39.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:34:39.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:34:39.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:34:39.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:34:39.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:34:39.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:34:39.869 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.869 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.869 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.870 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:34:39.870 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:39.870 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 40.772298255999999, 2024-04-24T14:34:39.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:34:39.870 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.870 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:39.870 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:39.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:39.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:39.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:39.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:34:39.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:34:39.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:34:39.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:34:39.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:34:39.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:34:39.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:34:39.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:34:39.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:34:39.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:34:39.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:39.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:34:39.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:34:39.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:34:39.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:34:39.875 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.875 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.875 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.875 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:39.875 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:39.875 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.928147508000002, 2024-04-24T14:34:39.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:34:39.875 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.875 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:39.876 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:39.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:39.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:39.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:39.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:34:39.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:34:39.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:34:39.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:34:39.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:34:39.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:34:39.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:34:39.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:34:39.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:34:39.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:34:39.879 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.879 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.879 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.879 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:5 3.2 3.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:34:39.879 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:39.879 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.599909866000001, 2024-04-24T14:34:39.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043717790000000001, 2024-04-24T14:34:39.879 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.879 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-24T14:34:39.879 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:39.880 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:34:39.880 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:34:39.880 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 5 2024-04-24T14:34:39.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.880 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.880 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:39.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.880 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:39.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207678+0000", 2024-04-24T14:34:39.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999962 2024-04-24T14:34:39.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207700+0000", 2024-04-24T14:34:39.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1313000000000001e-05 2024-04-24T14:34:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207757+0000", 2024-04-24T14:34:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7499e-05 2024-04-24T14:34:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207779+0000", 2024-04-24T14:34:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1478999999999999e-05 2024-04-24T14:34:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.212054+0000", 2024-04-24T14:34:39.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042750849999999996 2024-04-24T14:34:39.883 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.883 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.883 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.883 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:39.883 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:39.883 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.704384508, 2024-04-24T14:34:39.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044084859999999997, 2024-04-24T14:34:39.883 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.883 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:39.884 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:39.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:39.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103206+0000", 2024-04-24T14:34:39.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:39.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104731+0000", 2024-04-24T14:34:39.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001524755 2024-04-24T14:34:39.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104759+0000", 2024-04-24T14:34:39.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7973000000000001e-05 2024-04-24T14:34:39.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104807+0000", 2024-04-24T14:34:39.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8112000000000003e-05 2024-04-24T14:34:39.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104907+0000", 2024-04-24T14:34:39.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100317 2024-04-24T14:34:39.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104918+0000", 2024-04-24T14:34:39.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747999999999999e-05 2024-04-24T14:34:39.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.107616+0000", 2024-04-24T14:34:39.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026977559999999999 2024-04-24T14:34:39.888 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.888 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.888 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.888 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(3.2 83/82 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:39.888 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:39.888 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.697990131000001, 2024-04-24T14:34:39.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041425760000000002, 2024-04-24T14:34:39.888 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.888 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:39.888 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:39.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:39.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109600+0000", 2024-04-24T14:34:39.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:39.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111210+0000", 2024-04-24T14:34:39.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016097279999999999 2024-04-24T14:34:39.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111237+0000", 2024-04-24T14:34:39.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7189999999999999e-05 2024-04-24T14:34:39.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111251+0000", 2024-04-24T14:34:39.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.365e-05 2024-04-24T14:34:39.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111367+0000", 2024-04-24T14:34:39.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000115703 2024-04-24T14:34:39.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111378+0000", 2024-04-24T14:34:39.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1873e-05 2024-04-24T14:34:39.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.113744+0000", 2024-04-24T14:34:39.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023657069999999999 2024-04-24T14:34:39.892 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.892 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.892 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.893 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:39.893 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:39.893 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.681089555, 2024-04-24T14:34:39.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046193859999999996, 2024-04-24T14:34:39.893 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.893 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:39.893 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:39.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:39.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126501+0000", 2024-04-24T14:34:39.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:39.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128169+0000", 2024-04-24T14:34:39.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016686229999999999 2024-04-24T14:34:39.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128202+0000", 2024-04-24T14:34:39.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2539000000000001e-05 2024-04-24T14:34:39.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128222+0000", 2024-04-24T14:34:39.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0149e-05 2024-04-24T14:34:39.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128375+0000", 2024-04-24T14:34:39.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000153459 2024-04-24T14:34:39.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128399+0000", 2024-04-24T14:34:39.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3393000000000001e-05 2024-04-24T14:34:39.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.131122+0000", 2024-04-24T14:34:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027227140000000002 2024-04-24T14:34:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 49'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@49'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 21.659308175, 2024-04-24T14:34:39.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038414529999999999, 2024-04-24T14:34:39.898 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:39.898 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:39.898 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:39.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.898 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:39.898 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:39.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.898 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:39.898 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:39.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:39.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:39.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148282+0000", 2024-04-24T14:34:39.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:39.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:39.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149875+0000", 2024-04-24T14:34:39.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015929900000000001 2024-04-24T14:34:39.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:39.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149908+0000", 2024-04-24T14:34:39.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3102000000000003e-05 2024-04-24T14:34:39.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:39.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149941+0000", 2024-04-24T14:34:39.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2480000000000001e-05 2024-04-24T14:34:39.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:39.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150031+0000", 2024-04-24T14:34:39.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0394000000000002e-05 2024-04-24T14:34:39.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:39.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150043+0000", 2024-04-24T14:34:39.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1967000000000001e-05 2024-04-24T14:34:39.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:39.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:39.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:39.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.152125+0000", 2024-04-24T14:34:39.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020817029999999999 2024-04-24T14:34:39.901 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.902 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.902 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.902 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:39.902 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:39.902 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:39.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:39.904 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:39.909+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:40.004 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:40.006+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:40.011 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:40.012 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:40.012 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:40.012 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:40.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:40.105 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:40.109+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:40.124 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:40.124 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:40.124 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:40.124 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:40.125 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:40.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:40.163 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 1] 2024-04-24T14:34:40.163 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:34:40.163 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4] 2024-04-24T14:34:40.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-04-24T14:34:40.206 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:40.210+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:40.231 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:40.232 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:40.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:40.232 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:40.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.232 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:40.232 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:40.232 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.354939139999999, 2024-04-24T14:34:40.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:34:40.232 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.232 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.232 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.232 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:40.232 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:34:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:34:40.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:40.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:34:40.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:34:40.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:34:40.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:34:40.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:34:40.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:34:40.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:34:40.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:34:40.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:34:40.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:34:40.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:34:40.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:34:40.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:34:40.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:34:40.238 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.238 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.238 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.238 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:40.238 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:40.238 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.354910154999999, 2024-04-24T14:34:40.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:34:40.238 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.238 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.238 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.238 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:34:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:40.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:34:40.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:34:40.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:34:40.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:34:40.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:34:40.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:34:40.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:34:40.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:34:40.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:40.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:34:40.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:34:40.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:34:40.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:34:40.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:40.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:34:40.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:34:40.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:34:40.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:34:40.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:34:40.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:34:40.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:34:40.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:34:40.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.244 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.244 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:40.244 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:40.244 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.354891670000001, 2024-04-24T14:34:40.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:34:40.244 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.244 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:40.245 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:40.245 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:34:40.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.245 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:40.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:40.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:34:40.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:34:40.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:34:40.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:34:40.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:34:40.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:34:40.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:34:40.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:34:40.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:34:40.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:40.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:34:40.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:34:40.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:34:40.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:34:40.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:40.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:34:40.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:34:40.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:34:40.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:34:40.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:34:40.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:34:40.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:34:40.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:34:40.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.250 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.250 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:40.250 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:40.250 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.354876388999998, 2024-04-24T14:34:40.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:34:40.250 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.250 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.251 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.251 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:40.251 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:40.251 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:34:40.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.251 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:40.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:40.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:40.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:34:40.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:34:40.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:34:40.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:34:40.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:34:40.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:34:40.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:34:40.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:34:40.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:40.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:34:40.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:34:40.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:34:40.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:34:40.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:40.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:34:40.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:34:40.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:34:40.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:34:40.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:34:40.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:34:40.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:34:40.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:34:40.256 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.256 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.256 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.256 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:40.256 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:40.257 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.354860608000003, 2024-04-24T14:34:40.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:34:40.257 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.257 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.257 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.257 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:40.257 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:40.257 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:34:40.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.257 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:40.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:40.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:40.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:34:40.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:34:40.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:34:40.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:34:40.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:34:40.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:34:40.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:40.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:34:40.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:34:40.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:34:40.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:34:40.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:34:40.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:34:40.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:34:40.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:34:40.262 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.262 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.262 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.262 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:40.262 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:40.262 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.35484632, 2024-04-24T14:34:40.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:34:40.263 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.263 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.263 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.263 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:40.263 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:40.263 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:34:40.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.263 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:40.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:40.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:40.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:34:40.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:34:40.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:34:40.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:34:40.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:34:40.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:34:40.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:34:40.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:34:40.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:40.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:34:40.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:34:40.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:34:40.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:34:40.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:40.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:34:40.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:34:40.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:34:40.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:34:40.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:34:40.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:34:40.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:34:40.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:34:40.268 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.268 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.268 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.268 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:40.268 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:40.268 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.354826568, 2024-04-24T14:34:40.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:34:40.269 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.269 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.269 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.269 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:40.269 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:40.269 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:34:40.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.269 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:40.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:40.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:40.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:34:40.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:34:40.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:34:40.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:34:40.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:34:40.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:34:40.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:34:40.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:34:40.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:40.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:34:40.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:34:40.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:34:40.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:34:40.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:40.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:34:40.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:34:40.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:34:40.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:34:40.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:34:40.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:34:40.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:34:40.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:34:40.274 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.274 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.274 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.274 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:40.274 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:40.274 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.354809121000002, 2024-04-24T14:34:40.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:34:40.275 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.275 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.275 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.275 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:40.275 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:40.275 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:34:40.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.275 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:40.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:40.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:40.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:34:40.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:34:40.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:34:40.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:34:40.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:34:40.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:34:40.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:34:40.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:34:40.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:40.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:34:40.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:34:40.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:34:40.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:34:40.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:40.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:34:40.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:34:40.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:34:40.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:34:40.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:34:40.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:34:40.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:34:40.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:34:40.280 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.280 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.280 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.280 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:100 3.2 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.280 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:40.280 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.480754306, 2024-04-24T14:34:40.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009078015, 2024-04-24T14:34:40.281 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.281 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.281 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 100 2024-04-24T14:34:40.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.281 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:40.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:40.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754019+0000", 2024-04-24T14:34:40.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754025+0000", 2024-04-24T14:34:40.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2110000000000003e-06 2024-04-24T14:34:40.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754037+0000", 2024-04-24T14:34:40.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1868999999999999e-05 2024-04-24T14:34:40.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754046+0000", 2024-04-24T14:34:40.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7430000000000008e-06 2024-04-24T14:34:40.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754095+0000", 2024-04-24T14:34:40.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8924000000000001e-05 2024-04-24T14:34:40.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754153+0000", 2024-04-24T14:34:40.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8165000000000003e-05 2024-04-24T14:34:40.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:40.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754262+0000", 2024-04-24T14:34:40.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108694 2024-04-24T14:34:40.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.762999+0000", 2024-04-24T14:34:40.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087372590000000007 2024-04-24T14:34:40.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763027+0000", 2024-04-24T14:34:40.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8467e-05 2024-04-24T14:34:40.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763070+0000", 2024-04-24T14:34:40.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2236e-05 2024-04-24T14:34:40.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763098+0000", 2024-04-24T14:34:40.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8561999999999999e-05 2024-04-24T14:34:40.286 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.286 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.286 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.286 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:101 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [delete] snapc 7=[7,6,4] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.286 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:40.286 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.27554664, 2024-04-24T14:34:40.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012894743, 2024-04-24T14:34:40.286 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 101 2024-04-24T14:34:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959227+0000", 2024-04-24T14:34:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959233+0000", 2024-04-24T14:34:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.857e-06 2024-04-24T14:34:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959249+0000", 2024-04-24T14:34:40.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6155999999999999e-05 2024-04-24T14:34:40.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959257+0000", 2024-04-24T14:34:40.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7440000000000007e-06 2024-04-24T14:34:40.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959790+0000", 2024-04-24T14:34:40.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053381100000000003 2024-04-24T14:34:40.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960206+0000", 2024-04-24T14:34:40.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041549700000000002 2024-04-24T14:34:40.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:40.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960744+0000", 2024-04-24T14:34:40.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053762199999999997 2024-04-24T14:34:40.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971963+0000", 2024-04-24T14:34:40.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011219816000000001 2024-04-24T14:34:40.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972021+0000", 2024-04-24T14:34:40.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7855000000000001e-05 2024-04-24T14:34:40.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972073+0000", 2024-04-24T14:34:40.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2196e-05 2024-04-24T14:34:40.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972123+0000", 2024-04-24T14:34:40.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9311999999999998e-05 2024-04-24T14:34:40.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.292 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.292 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:102 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.292 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:40.292 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.275506613999999, 2024-04-24T14:34:40.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012939713, 2024-04-24T14:34:40.292 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.292 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 102 2024-04-24T14:34:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:40.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959267+0000", 2024-04-24T14:34:40.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959274+0000", 2024-04-24T14:34:40.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.593e-06 2024-04-24T14:34:40.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959291+0000", 2024-04-24T14:34:40.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6387000000000001e-05 2024-04-24T14:34:40.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959301+0000", 2024-04-24T14:34:40.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0023e-05 2024-04-24T14:34:40.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969594+0000", 2024-04-24T14:34:40.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010293182 2024-04-24T14:34:40.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969902+0000", 2024-04-24T14:34:40.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030835299999999998 2024-04-24T14:34:40.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:40.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970408+0000", 2024-04-24T14:34:40.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050588100000000002 2024-04-24T14:34:40.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972009+0000", 2024-04-24T14:34:40.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016011210000000001 2024-04-24T14:34:40.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972132+0000", 2024-04-24T14:34:40.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122521 2024-04-24T14:34:40.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972174+0000", 2024-04-24T14:34:40.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2242e-05 2024-04-24T14:34:40.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972208+0000", 2024-04-24T14:34:40.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3522999999999997e-05 2024-04-24T14:34:40.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.298 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.298 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:104 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.298 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:40.298 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.2748954, 2024-04-24T14:34:40.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014589062, 2024-04-24T14:34:40.298 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.298 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.299 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 104 2024-04-24T14:34:40.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.299 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:40.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:40.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959877+0000", 2024-04-24T14:34:40.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:40.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959890+0000", 2024-04-24T14:34:40.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2537e-05 2024-04-24T14:34:40.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959909+0000", 2024-04-24T14:34:40.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9340000000000001e-05 2024-04-24T14:34:40.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959922+0000", 2024-04-24T14:34:40.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2882e-05 2024-04-24T14:34:40.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970900+0000", 2024-04-24T14:34:40.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010977736 2024-04-24T14:34:40.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971039+0000", 2024-04-24T14:34:40.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139593 2024-04-24T14:34:40.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:40.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971453+0000", 2024-04-24T14:34:40.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041351099999999998 2024-04-24T14:34:40.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974044+0000", 2024-04-24T14:34:40.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025908289999999998 2024-04-24T14:34:40.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974377+0000", 2024-04-24T14:34:40.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033310399999999997 2024-04-24T14:34:40.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974433+0000", 2024-04-24T14:34:40.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6166999999999999e-05 2024-04-24T14:34:40.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974468+0000", 2024-04-24T14:34:40.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5336000000000002e-05 2024-04-24T14:34:40.304 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.304 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.304 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.304 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:106 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.304 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:40.304 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.274836067000001, 2024-04-24T14:34:40.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017192064999999999, 2024-04-24T14:34:40.304 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.304 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.305 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.305 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.305 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.305 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 106 2024-04-24T14:34:40.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.305 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:40.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:40.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959938+0000", 2024-04-24T14:34:40.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959943+0000", 2024-04-24T14:34:40.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.677e-06 2024-04-24T14:34:40.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959953+0000", 2024-04-24T14:34:40.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8069999999999994e-06 2024-04-24T14:34:40.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:34:40.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0560000000000001e-06 2024-04-24T14:34:40.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972232+0000", 2024-04-24T14:34:40.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012272460000000001 2024-04-24T14:34:40.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972304+0000", 2024-04-24T14:34:40.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2630000000000001e-05 2024-04-24T14:34:40.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:40.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972548+0000", 2024-04-24T14:34:40.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000244047 2024-04-24T14:34:40.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976900+0000", 2024-04-24T14:34:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043514069999999998 2024-04-24T14:34:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977076+0000", 2024-04-24T14:34:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176281 2024-04-24T14:34:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977106+0000", 2024-04-24T14:34:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0015e-05 2024-04-24T14:34:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977131+0000", 2024-04-24T14:34:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4669e-05 2024-04-24T14:34:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:114 3.2 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.251514903, 2024-04-24T14:34:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090880089999999993, 2024-04-24T14:34:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 114 2024-04-24T14:34:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983259+0000", 2024-04-24T14:34:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983264+0000", 2024-04-24T14:34:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9130000000000002e-06 2024-04-24T14:34:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983277+0000", 2024-04-24T14:34:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3723999999999999e-05 2024-04-24T14:34:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983282+0000", 2024-04-24T14:34:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0429999999999998e-06 2024-04-24T14:34:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983339+0000", 2024-04-24T14:34:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6323000000000003e-05 2024-04-24T14:34:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983464+0000", 2024-04-24T14:34:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012554299999999999 2024-04-24T14:34:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983693+0000", 2024-04-24T14:34:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000228848 2024-04-24T14:34:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.986978+0000", 2024-04-24T14:34:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032852810000000001 2024-04-24T14:34:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992288+0000", 2024-04-24T14:34:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053095369999999996 2024-04-24T14:34:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992322+0000", 2024-04-24T14:34:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3683999999999998e-05 2024-04-24T14:34:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992348+0000", 2024-04-24T14:34:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6157999999999999e-05 2024-04-24T14:34:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:121 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.239458804, 2024-04-24T14:34:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098798159999999996, 2024-04-24T14:34:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 121 2024-04-24T14:34:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995315+0000", 2024-04-24T14:34:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995320+0000", 2024-04-24T14:34:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0239999999999997e-06 2024-04-24T14:34:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995328+0000", 2024-04-24T14:34:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7800000000000001e-06 2024-04-24T14:34:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995334+0000", 2024-04-24T14:34:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2330000000000003e-06 2024-04-24T14:34:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998810+0000", 2024-04-24T14:34:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034756090000000002 2024-04-24T14:34:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998870+0000", 2024-04-24T14:34:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0625999999999999e-05 2024-04-24T14:34:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.999091+0000", 2024-04-24T14:34:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000220631 2024-04-24T14:34:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.001389+0000", 2024-04-24T14:34:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022977549999999998 2024-04-24T14:34:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005097+0000", 2024-04-24T14:34:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037085439999999998 2024-04-24T14:34:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005154+0000", 2024-04-24T14:34:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6208000000000001e-05 2024-04-24T14:34:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005196+0000", 2024-04-24T14:34:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2052000000000003e-05 2024-04-24T14:34:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:127 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.229157288, 2024-04-24T14:34:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013414918, 2024-04-24T14:34:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2024-04-24T14:34:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005616+0000", 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005622+0000", 2024-04-24T14:34:40.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2020000000000003e-06 2024-04-24T14:34:40.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005632+0000", 2024-04-24T14:34:40.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2024-04-24T14:34:40.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005637+0000", 2024-04-24T14:34:40.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7439999999999998e-06 2024-04-24T14:34:40.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005947+0000", 2024-04-24T14:34:40.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000310792 2024-04-24T14:34:40.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006059+0000", 2024-04-24T14:34:40.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011192399999999999 2024-04-24T14:34:40.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:40.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006503+0000", 2024-04-24T14:34:40.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000443394 2024-04-24T14:34:40.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018864+0000", 2024-04-24T14:34:40.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012361162 2024-04-24T14:34:40.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018877+0000", 2024-04-24T14:34:40.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3497e-05 2024-04-24T14:34:40.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018965+0000", 2024-04-24T14:34:40.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7476000000000001e-05 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019032+0000", 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7469000000000001e-05 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:129 3.2 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.229017055, 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010059754000000001, 2024-04-24T14:34:40.328 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.329 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.329 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.329 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.329 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.329 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 129 2024-04-24T14:34:40.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.329 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:40.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:40.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005757+0000", 2024-04-24T14:34:40.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005760+0000", 2024-04-24T14:34:40.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2430000000000001e-06 2024-04-24T14:34:40.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005769+0000", 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4810000000000006e-06 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005773+0000", 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005840+0000", 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6934999999999995e-05 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005961+0000", 2024-04-24T14:34:40.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012153 2024-04-24T14:34:40.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:40.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006181+0000", 2024-04-24T14:34:40.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021947500000000001 2024-04-24T14:34:40.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015520+0000", 2024-04-24T14:34:40.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093393049999999991 2024-04-24T14:34:40.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015733+0000", 2024-04-24T14:34:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021355699999999999 2024-04-24T14:34:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015773+0000", 2024-04-24T14:34:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9554000000000001e-05 2024-04-24T14:34:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015817+0000", 2024-04-24T14:34:40.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4325e-05 2024-04-24T14:34:40.334 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.334 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.334 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.334 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:131 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.334 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:40.334 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.228739173999999, 2024-04-24T14:34:40.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014251976, 2024-04-24T14:34:40.334 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.334 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.334 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.334 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.334 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131 2024-04-24T14:34:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006034+0000", 2024-04-24T14:34:40.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:40.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006042+0000", 2024-04-24T14:34:40.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2970000000000004e-06 2024-04-24T14:34:40.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006061+0000", 2024-04-24T14:34:40.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8505999999999999e-05 2024-04-24T14:34:40.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006071+0000", 2024-04-24T14:34:40.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0409e-05 2024-04-24T14:34:40.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007899+0000", 2024-04-24T14:34:40.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018274789999999999 2024-04-24T14:34:40.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008281+0000", 2024-04-24T14:34:40.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038244700000000001 2024-04-24T14:34:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008528+0000", 2024-04-24T14:34:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024723900000000001 2024-04-24T14:34:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019070+0000", 2024-04-24T14:34:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010541867999999999 2024-04-24T14:34:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020099+0000", 2024-04-24T14:34:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00102852 2024-04-24T14:34:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020226+0000", 2024-04-24T14:34:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012744300000000001 2024-04-24T14:34:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020287+0000", 2024-04-24T14:34:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1253000000000005e-05 2024-04-24T14:34:40.339 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.340 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.340 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.340 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:132 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.340 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:40.340 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.228685321, 2024-04-24T14:34:40.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015917981000000001, 2024-04-24T14:34:40.340 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.340 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.340 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.340 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.340 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.341 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 132 2024-04-24T14:34:40.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.341 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:40.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:40.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:40.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006093+0000", 2024-04-24T14:34:40.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3819999999999997e-06 2024-04-24T14:34:40.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006103+0000", 2024-04-24T14:34:40.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0352e-05 2024-04-24T14:34:40.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006110+0000", 2024-04-24T14:34:40.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3910000000000003e-06 2024-04-24T14:34:40.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015586+0000", 2024-04-24T14:34:40.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094759119999999995 2024-04-24T14:34:40.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015730+0000", 2024-04-24T14:34:40.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014487199999999999 2024-04-24T14:34:40.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:40.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015886+0000", 2024-04-24T14:34:40.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015573799999999999 2024-04-24T14:34:40.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020306+0000", 2024-04-24T14:34:40.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044199399999999998 2024-04-24T14:34:40.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021887+0000", 2024-04-24T14:34:40.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015809400000000001 2024-04-24T14:34:40.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021942+0000", 2024-04-24T14:34:40.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5071000000000002e-05 2024-04-24T14:34:40.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.022007+0000", 2024-04-24T14:34:40.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5110000000000005e-05 2024-04-24T14:34:40.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.345 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.346 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.346 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.5324.0:127 3.0 e87/69 ondisk, result = 0)", 2024-04-24T14:34:40.346 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:40.346 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.226976253, 2024-04-24T14:34:40.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011245125, 2024-04-24T14:34:40.346 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.346 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.346 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:40.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:40.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:40.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007804+0000", 2024-04-24T14:34:40.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6610000000000003e-06 2024-04-24T14:34:40.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007817+0000", 2024-04-24T14:34:40.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3030999999999999e-05 2024-04-24T14:34:40.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007832+0000", 2024-04-24T14:34:40.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.47e-05 2024-04-24T14:34:40.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018843+0000", 2024-04-24T14:34:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011010647 2024-04-24T14:34:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018853+0000", 2024-04-24T14:34:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319000000000001e-05 2024-04-24T14:34:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019043+0000", 2024-04-24T14:34:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019055600000000001 2024-04-24T14:34:40.350 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.350 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.350 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.350 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.350 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.350 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:40.351 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:40.352 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:40.357+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:40.452 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:40.452 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:40.452 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:40.453 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:40.454+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:40.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:40.477 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-04-24T14:34:40.478+0000 7ff9c25de700 -1 mon.a@0(leader).osd e95 definitely_dead 0 2024-04-24T14:34:40.553 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:40.554 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:40.554 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:40.554 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:40.554 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:40.554 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:40.558+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:40.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:40.655 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:40.661+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:40.713 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.960125798999997, 2024-04-24T14:34:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:34:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:34:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:34:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:34:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:34:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:34:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:34:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:34:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:34:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:34:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:34:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:34:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:34:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:34:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:34:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:34:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:34:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:34:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:34:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:34:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:34:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:34:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:34:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.720 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:40.720 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:40.720 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.960074964999997, 2024-04-24T14:34:40.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:34:40.720 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.720 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.720 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.720 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:40.720 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:40.720 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:34:40.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.720 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:40.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:40.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:40.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:34:40.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:34:40.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:34:40.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:34:40.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:34:40.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:34:40.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:34:40.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:34:40.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:40.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:34:40.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:34:40.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:34:40.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:34:40.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:40.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:34:40.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:34:40.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:34:40.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:34:40.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:34:40.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:34:40.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:34:40.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:34:40.725 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.725 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.725 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.726 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:40.726 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:40.726 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.960038658000002, 2024-04-24T14:34:40.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:34:40.726 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.726 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.726 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:40.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:40.727 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:34:40.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.727 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:40.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:40.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:34:40.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:34:40.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:34:40.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:34:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:34:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:34:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:34:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:34:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:34:40.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:40.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:34:40.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:34:40.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:34:40.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:34:40.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:40.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:34:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:34:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:34:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:34:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:34:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:34:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.956627408000003, 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.732 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:40.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:40.733 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:34:40.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.733 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:40.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:40.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:34:40.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:40.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:34:40.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:34:40.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:34:40.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:34:40.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:34:40.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:34:40.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:34:40.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:34:40.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:40.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:34:40.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:34:40.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:34:40.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:34:40.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:40.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:34:40.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:34:40.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:34:40.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:34:40.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:34:40.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:34:40.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:34:40.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:34:40.738 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.738 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.738 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.738 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:40.738 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:40.738 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.956565730999998, 2024-04-24T14:34:40.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:34:40.738 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.738 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.738 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.739 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:40.739 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:40.739 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:34:40.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.739 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:40.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:40.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:34:40.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:34:40.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:34:40.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:34:40.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:34:40.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:34:40.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:34:40.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:34:40.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:34:40.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:34:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:34:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:34:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:34:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:40.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:34:40.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:34:40.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:40.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:34:40.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:34:40.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:34:40.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:34:40.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:34:40.744 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.744 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.744 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.744 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:40.744 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:40.744 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.956531155, 2024-04-24T14:34:40.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:34:40.744 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.744 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:40.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:40.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:34:40.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:34:40.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:34:40.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:34:40.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:34:40.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:34:40.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:34:40.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:34:40.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:40.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:34:40.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:34:40.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:34:40.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:34:40.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:40.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:34:40.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:34:40.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:34:40.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:34:40.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:34:40.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:34:40.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:34:40.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:34:40.750 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.750 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.750 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.750 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:40.750 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:40.750 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.956501815999999, 2024-04-24T14:34:40.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:34:40.750 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.750 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.750 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.751 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:40.751 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:40.751 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:34:40.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.751 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:40.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:34:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:34:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:34:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:34:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:34:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:34:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:34:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:34:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:34:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:40.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:34:40.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:34:40.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:34:40.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:34:40.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:40.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:34:40.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:34:40.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:34:40.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:34:40.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:34:40.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:34:40.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:34:40.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:34:40.756 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.756 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.756 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.756 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:40.756 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:40.756 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.956425234000001, 2024-04-24T14:34:40.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:34:40.756 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.756 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.757 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:40.757 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:40.757 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:34:40.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.757 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:40.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:40.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:34:40.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:34:40.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:34:40.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:34:40.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:34:40.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:34:40.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:34:40.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:34:40.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:34:40.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:40.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:34:40.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:34:40.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:34:40.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:34:40.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:40.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:34:40.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:34:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:34:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:34:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:34:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:34:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:34:40.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:34:40.762 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.762 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.762 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.762 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:40.762 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:40.762 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.956377689, 2024-04-24T14:34:40.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:34:40.762 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.762 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:40.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:40.763 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:34:40.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.763 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:40.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:34:40.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:34:40.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:34:40.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:34:40.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:34:40.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:34:40.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:40.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:34:40.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:34:40.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:34:40.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:34:40.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:40.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:34:40.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:34:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:34:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:34:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:34:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:34:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.956337046999998, 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:34:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:34:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:34:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:34:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:34:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:34:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:34:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:34:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:34:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:34:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:34:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:34:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:34:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:34:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:34:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:34:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.956319936, 2024-04-24T14:34:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:34:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:34:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:34:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:34:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:34:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:34:40.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:34:40.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:34:40.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:34:40.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:34:40.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:34:40.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:34:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:34:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:34:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:34:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:34:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:34:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:34:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:34:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:34:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:34:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:34:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:34:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:103 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.756753823, 2024-04-24T14:34:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032646351999999997, 2024-04-24T14:34:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 103 2024-04-24T14:34:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959674+0000", 2024-04-24T14:34:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959688+0000", 2024-04-24T14:34:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2024-04-24T14:34:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959722+0000", 2024-04-24T14:34:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3936000000000002e-05 2024-04-24T14:34:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959735+0000", 2024-04-24T14:34:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2926e-05 2024-04-24T14:34:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959921+0000", 2024-04-24T14:34:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185601 2024-04-24T14:34:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960324+0000", 2024-04-24T14:34:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040356700000000002 2024-04-24T14:34:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960839+0000", 2024-04-24T14:34:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051410599999999996 2024-04-24T14:34:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.975634+0000", 2024-04-24T14:34:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014795770999999999 2024-04-24T14:34:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992113+0000", 2024-04-24T14:34:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016478982999999999 2024-04-24T14:34:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992228+0000", 2024-04-24T14:34:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114295 2024-04-24T14:34:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992323+0000", 2024-04-24T14:34:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5344000000000001e-05 2024-04-24T14:34:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:105 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.756504045, 2024-04-24T14:34:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032581757000000003, 2024-04-24T14:34:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105 2024-04-24T14:34:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959925+0000", 2024-04-24T14:34:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959937+0000", 2024-04-24T14:34:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2391999999999999e-05 2024-04-24T14:34:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:34:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2535999999999999e-05 2024-04-24T14:34:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959970+0000", 2024-04-24T14:34:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1039999999999999e-05 2024-04-24T14:34:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969531+0000", 2024-04-24T14:34:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095603949999999993 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969669+0000", 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013806499999999999 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970010+0000", 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034062200000000001 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976983+0000", 2024-04-24T14:34:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069735600000000002 2024-04-24T14:34:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992413+0000", 2024-04-24T14:34:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015429841999999999 2024-04-24T14:34:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992457+0000", 2024-04-24T14:34:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4169999999999999e-05 2024-04-24T14:34:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992508+0000", 2024-04-24T14:34:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0995000000000002e-05 2024-04-24T14:34:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:113 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.736277754, 2024-04-24T14:34:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014793863000000001, 2024-04-24T14:34:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 113 2024-04-24T14:34:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980152+0000", 2024-04-24T14:34:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980156+0000", 2024-04-24T14:34:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.619e-06 2024-04-24T14:34:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980163+0000", 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7070000000000003e-06 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980173+0000", 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0957e-05 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983381+0000", 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032079930000000001 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983628+0000", 2024-04-24T14:34:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024665700000000001 2024-04-24T14:34:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983964+0000", 2024-04-24T14:34:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000336332 2024-04-24T14:34:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992694+0000", 2024-04-24T14:34:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087297009999999994 2024-04-24T14:34:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994878+0000", 2024-04-24T14:34:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021841439999999998 2024-04-24T14:34:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994917+0000", 2024-04-24T14:34:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8544999999999998e-05 2024-04-24T14:34:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994947+0000", 2024-04-24T14:34:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9705000000000001e-05 2024-04-24T14:34:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:130 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.710584319000001, 2024-04-24T14:34:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014646289999999999, 2024-04-24T14:34:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 130 2024-04-24T14:34:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005844+0000", 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005853+0000", 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1319999999999994e-06 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005870+0000", 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7448999999999998e-05 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005876+0000", 2024-04-24T14:34:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3319999999999997e-06 2024-04-24T14:34:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005913+0000", 2024-04-24T14:34:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6835999999999998e-05 2024-04-24T14:34:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006017+0000", 2024-04-24T14:34:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010358500000000001 2024-04-24T14:34:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006139+0000", 2024-04-24T14:34:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001227 2024-04-24T14:34:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.010194+0000", 2024-04-24T14:34:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040549380000000001 2024-04-24T14:34:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020365+0000", 2024-04-24T14:34:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010170983 2024-04-24T14:34:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020428+0000", 2024-04-24T14:34:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2457000000000002e-05 2024-04-24T14:34:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020492+0000", 2024-04-24T14:34:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4592999999999996e-05 2024-04-24T14:34:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:155 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1601536~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.676923532, 2024-04-24T14:34:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015144391, 2024-04-24T14:34:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 155 2024-04-24T14:34:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039506+0000", 2024-04-24T14:34:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039766+0000", 2024-04-24T14:34:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025994400000000002 2024-04-24T14:34:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039780+0000", 2024-04-24T14:34:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3366e-05 2024-04-24T14:34:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039787+0000", 2024-04-24T14:34:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4359999999999996e-06 2024-04-24T14:34:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.050886+0000", 2024-04-24T14:34:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011098993 2024-04-24T14:34:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051026+0000", 2024-04-24T14:34:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001395 2024-04-24T14:34:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051211+0000", 2024-04-24T14:34:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018547099999999999 2024-04-24T14:34:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054490+0000", 2024-04-24T14:34:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032785499999999999 2024-04-24T14:34:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054505+0000", 2024-04-24T14:34:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5906999999999999e-05 2024-04-24T14:34:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054594+0000", 2024-04-24T14:34:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8817e-05 2024-04-24T14:34:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054651+0000", 2024-04-24T14:34:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7040999999999998e-05 2024-04-24T14:34:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:156 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1732608~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.676632826000001, 2024-04-24T14:34:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016895194999999998, 2024-04-24T14:34:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 156 2024-04-24T14:34:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039797+0000", 2024-04-24T14:34:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040065+0000", 2024-04-24T14:34:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000268361 2024-04-24T14:34:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040086+0000", 2024-04-24T14:34:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1024000000000001e-05 2024-04-24T14:34:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040094+0000", 2024-04-24T14:34:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9379999999999992e-06 2024-04-24T14:34:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051936+0000", 2024-04-24T14:34:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011842518 2024-04-24T14:34:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052062+0000", 2024-04-24T14:34:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125482 2024-04-24T14:34:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052206+0000", 2024-04-24T14:34:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143953 2024-04-24T14:34:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054715+0000", 2024-04-24T14:34:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025088240000000002 2024-04-24T14:34:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056534+0000", 2024-04-24T14:34:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001819436 2024-04-24T14:34:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056610+0000", 2024-04-24T14:34:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5383000000000007e-05 2024-04-24T14:34:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056693+0000", 2024-04-24T14:34:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3283999999999997e-05 2024-04-24T14:34:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:157 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1863680~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.676325447, 2024-04-24T14:34:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016763674999999999, 2024-04-24T14:34:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157 2024-04-24T14:34:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040104+0000", 2024-04-24T14:34:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040244+0000", 2024-04-24T14:34:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014035499999999999 2024-04-24T14:34:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040255+0000", 2024-04-24T14:34:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0135999999999999e-05 2024-04-24T14:34:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040260+0000", 2024-04-24T14:34:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7919999999999998e-06 2024-04-24T14:34:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053406+0000", 2024-04-24T14:34:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013146023999999999 2024-04-24T14:34:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053531+0000", 2024-04-24T14:34:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012471800000000001 2024-04-24T14:34:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053715+0000", 2024-04-24T14:34:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018395099999999999 2024-04-24T14:34:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056711+0000", 2024-04-24T14:34:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029956520000000001 2024-04-24T14:34:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056783+0000", 2024-04-24T14:34:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2010999999999999e-05 2024-04-24T14:34:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056831+0000", 2024-04-24T14:34:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8189999999999998e-05 2024-04-24T14:34:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056869+0000", 2024-04-24T14:34:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7812000000000003e-05 2024-04-24T14:34:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:158 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1994752~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.676163519999999, 2024-04-24T14:34:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016845334999999999, 2024-04-24T14:34:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2024-04-24T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040266+0000", 2024-04-24T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040379+0000", 2024-04-24T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112728 2024-04-24T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040387+0000", 2024-04-24T14:34:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8499999999999994e-06 2024-04-24T14:34:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040391+0000", 2024-04-24T14:34:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2719999999999999e-06 2024-04-24T14:34:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054803+0000", 2024-04-24T14:34:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014411548 2024-04-24T14:34:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054982+0000", 2024-04-24T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017941699999999999 2024-04-24T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.055167+0000", 2024-04-24T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018542100000000001 2024-04-24T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056718+0000", 2024-04-24T14:34:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015505670000000001 2024-04-24T14:34:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057020+0000", 2024-04-24T14:34:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030227500000000002 2024-04-24T14:34:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057070+0000", 2024-04-24T14:34:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9611000000000003e-05 2024-04-24T14:34:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057112+0000", 2024-04-24T14:34:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2373999999999998e-05 2024-04-24T14:34:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:159 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.670728532, 2024-04-24T14:34:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015930650000000001, 2024-04-24T14:34:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2024-04-24T14:34:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045701+0000", 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045707+0000", 2024-04-24T14:34:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.562e-06 2024-04-24T14:34:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045728+0000", 2024-04-24T14:34:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1109999999999999e-05 2024-04-24T14:34:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045738+0000", 2024-04-24T14:34:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0016e-05 2024-04-24T14:34:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056183+0000", 2024-04-24T14:34:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010444836000000001 2024-04-24T14:34:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056470+0000", 2024-04-24T14:34:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028696899999999998 2024-04-24T14:34:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057160+0000", 2024-04-24T14:34:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069004699999999995 2024-04-24T14:34:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057291+0000", 2024-04-24T14:34:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013096200000000001 2024-04-24T14:34:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.061633+0000", 2024-04-24T14:34:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004341414 2024-04-24T14:34:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:40.837 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:40.837 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:40.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:40.839 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:40.842+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:40.939 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:40.942+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:40.946 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:40.946 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:40.946 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:40.946 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:40.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:41.039 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:41.045+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:41.104 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:41.104 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:41.104 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:41.105 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:41.105 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:41.113 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:41.141 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:41.145+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:41.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T14:34:41.198 INFO:teuthology.orchestra.run.smithi090.stderr:marked out osd.1. 2024-04-24T14:34:41.225 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.75951397, 2024-04-24T14:34:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:34:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:34:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:41.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:41.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:34:41.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:41.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:34:41.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:34:41.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:34:41.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:34:41.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:34:41.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:34:41.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:34:41.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:34:41.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:34:41.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:34:41.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:34:41.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:34:41.230 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.230 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.230 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.230 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.230 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:41.230 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.384157266999999, 2024-04-24T14:34:41.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:34:41.230 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.231 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.231 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.231 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.231 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.231 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:34:41.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.231 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:41.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:41.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:34:41.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:41.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:34:41.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:34:41.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:34:41.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:34:41.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:34:41.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:41.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:34:41.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:34:41.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:34:41.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:34:41.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:34:41.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:34:41.234 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.234 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.235 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.235 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.235 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:41.235 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.382729113000003, 2024-04-24T14:34:41.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:34:41.235 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.235 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.235 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.235 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.235 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.235 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:34:41.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.236 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:41.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:41.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:34:41.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:41.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:34:41.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:34:41.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:34:41.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:34:41.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:34:41.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:34:41.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:34:41.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:34:41.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:34:41.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:34:41.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:34:41.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:34:41.239 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.239 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.239 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.239 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.240 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:41.240 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.382702023999997, 2024-04-24T14:34:41.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:34:41.240 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.240 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.240 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:34:41.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.240 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:41.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:41.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:34:41.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:41.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:34:41.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:34:41.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:34:41.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:34:41.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:34:41.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:34:41.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:34:41.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:34:41.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:34:41.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:34:41.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:34:41.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:34:41.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.244 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.244 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.244 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:41.244 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.382678057, 2024-04-24T14:34:41.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:34:41.244 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.244 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.245 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.245 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.245 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:34:41.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.245 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:41.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:34:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:34:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:34:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:34:41.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:34:41.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:34:41.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:34:41.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:34:41.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:34:41.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:34:41.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:34:41.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:34:41.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:34:41.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.248 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.249 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.249 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:41.249 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.368694077999997, 2024-04-24T14:34:41.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:34:41.249 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.249 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.249 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:34:41.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.249 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:41.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:41.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:34:41.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:41.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:34:41.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:34:41.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:34:41.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:34:41.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:34:41.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:34:41.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:34:41.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:34:41.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:41.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:34:41.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:34:41.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:34:41.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:34:41.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:41.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:34:41.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:34:41.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:34:41.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:34:41.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:34:41.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:34:41.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:34:41.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:34:41.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.254 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.255 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.255 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:41.255 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.368577766999998, 2024-04-24T14:34:41.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:34:41.255 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.255 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.255 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.255 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.255 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.255 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:34:41.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.256 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:41.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:41.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:34:41.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:41.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:34:41.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:34:41.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:34:41.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:34:41.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:34:41.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:34:41.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:34:41.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:34:41.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:41.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:34:41.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:34:41.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:34:41.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:34:41.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:41.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:34:41.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:34:41.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:34:41.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:34:41.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:34:41.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:34:41.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:34:41.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:34:41.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.368530108999998, 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:34:41.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.262 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:41.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:41.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:34:41.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:41.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:34:41.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:34:41.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:34:41.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:34:41.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:34:41.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:34:41.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:34:41.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:34:41.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:41.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:34:41.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:34:41.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:34:41.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:34:41.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:41.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:34:41.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:34:41.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:34:41.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:34:41.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:34:41.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:34:41.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.364492810999998, 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.268 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.268 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:34:41.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.268 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:41.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:41.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:34:41.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:41.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:34:41.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:34:41.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:41.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:34:41.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:34:41.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:34:41.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:34:41.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:34:41.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.364429051999998, 2024-04-24T14:34:41.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:41.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:41.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:34:41.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:41.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:34:41.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:34:41.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:34:41.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:41.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:34:41.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:34:41.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:34:41.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:34:41.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:41.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:34:41.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:34:41.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:34:41.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:34:41.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:34:41.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:34:41.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:34:41.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:34:41.279 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.279 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.279 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.280 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.280 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:41.280 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.364403562, 2024-04-24T14:34:41.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:34:41.280 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.280 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.280 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:34:41.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.280 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:41.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:34:41.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.286 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.286 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:41.286 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.364379812000003, 2024-04-24T14:34:41.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:34:41.286 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.286 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.286 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.286 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.286 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.286 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:34:41.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.286 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:34:41.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:34:41.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:41.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:34:41.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:34:41.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:34:41.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:34:41.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:34:41.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:34:41.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:34:41.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:34:41.291 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.291 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.291 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.364341299000003, 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:41.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:41.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:34:41.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:41.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:34:41.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:34:41.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:34:41.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:34:41.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:34:41.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:34:41.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:34:41.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:34:41.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:41.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:34:41.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:34:41.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:34:41.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:34:41.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:41.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:34:41.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:34:41.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:34:41.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:34:41.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:34:41.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:34:41.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:34:41.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:34:41.297 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.297 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.297 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.298 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.298 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:41.298 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.364313101999997, 2024-04-24T14:34:41.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:34:41.298 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.298 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.298 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.298 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.298 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.298 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:34:41.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.298 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:41.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:41.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:34:41.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:41.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:34:41.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:34:41.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:34:41.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:34:41.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:34:41.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:34:41.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:34:41.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:34:41.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:41.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:34:41.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:34:41.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:34:41.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:34:41.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:41.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:34:41.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:34:41.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:34:41.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:34:41.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:34:41.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:34:41.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:34:41.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:34:41.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.303 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.304 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.304 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:41.304 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.364291645000002, 2024-04-24T14:34:41.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:34:41.304 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.304 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.304 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:34:41.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.304 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:34:41.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:34:41.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:34:41.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:41.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:34:41.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:34:41.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:34:41.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:34:41.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:34:41.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:34:41.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:34:41.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:34:41.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.309 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.310 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:41.310 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:41.310 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.364272802999999, 2024-04-24T14:34:41.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:34:41.310 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.310 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:41.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:41.310 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:34:41.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.310 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:41.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:41.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:34:41.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.315 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.151440653999998, 2024-04-24T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:41.316 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:34:41.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:34:41.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:34:41.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:34:41.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:34:41.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:34:41.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:34:41.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.322 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.322 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:39 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1753088~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:41.322 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:41.322 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.211668238, 2024-04-24T14:34:41.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01139394, 2024-04-24T14:34:41.322 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.322 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:41.322 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.322 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:41.323 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:41.323 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 39 2024-04-24T14:34:41.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.323 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:41.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:41.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018967+0000", 2024-04-24T14:34:41.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:41.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019242+0000", 2024-04-24T14:34:41.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027504500000000002 2024-04-24T14:34:41.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019274+0000", 2024-04-24T14:34:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2472999999999999e-05 2024-04-24T14:34:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019288+0000", 2024-04-24T14:34:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3283e-05 2024-04-24T14:34:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026753+0000", 2024-04-24T14:34:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074651120000000003 2024-04-24T14:34:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026971+0000", 2024-04-24T14:34:41.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021810800000000001 2024-04-24T14:34:41.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:41.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.027211+0000", 2024-04-24T14:34:41.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023975 2024-04-24T14:34:41.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:41.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030200+0000", 2024-04-24T14:34:41.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029890709999999998 2024-04-24T14:34:41.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:41.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030252+0000", 2024-04-24T14:34:41.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1926999999999999e-05 2024-04-24T14:34:41.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:41.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030312+0000", 2024-04-24T14:34:41.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0602e-05 2024-04-24T14:34:41.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030363+0000", 2024-04-24T14:34:41.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0342e-05 2024-04-24T14:34:41.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.328 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.328 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:40 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1884160~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:41.328 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:41.328 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.211332263999999, 2024-04-24T14:34:41.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011199064, 2024-04-24T14:34:41.328 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.328 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:41.328 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.328 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:41.328 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:41.329 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 40 2024-04-24T14:34:41.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.329 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:41.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:41.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019303+0000", 2024-04-24T14:34:41.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:41.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019729+0000", 2024-04-24T14:34:41.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042620899999999998 2024-04-24T14:34:41.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019769+0000", 2024-04-24T14:34:41.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9847e-05 2024-04-24T14:34:41.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019797+0000", 2024-04-24T14:34:41.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8347000000000002e-05 2024-04-24T14:34:41.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028299+0000", 2024-04-24T14:34:41.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085010629999999997 2024-04-24T14:34:41.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028385+0000", 2024-04-24T14:34:41.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6824999999999996e-05 2024-04-24T14:34:41.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:41.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028510+0000", 2024-04-24T14:34:41.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012483000000000001 2024-04-24T14:34:41.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:41.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030410+0000", 2024-04-24T14:34:41.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001899612 2024-04-24T14:34:41.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:41.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030416+0000", 2024-04-24T14:34:41.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7699999999999998e-06 2024-04-24T14:34:41.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:41.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030461+0000", 2024-04-24T14:34:41.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5157999999999997e-05 2024-04-24T14:34:41.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030504+0000", 2024-04-24T14:34:41.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2988999999999998e-05 2024-04-24T14:34:41.333 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.334 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.334 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.334 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:41 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 2015232~81920] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:41.334 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:41.334 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.210823556999999, 2024-04-24T14:34:41.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012345257, 2024-04-24T14:34:41.334 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:41.334 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:41.334 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:41.334 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:41.334 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:41.335 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 41 2024-04-24T14:34:41.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.335 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:41.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:41.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:41.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:41.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:41.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:41.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:41.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019812+0000", 2024-04-24T14:34:41.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:41.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:41.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019975+0000", 2024-04-24T14:34:41.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016325700000000001 2024-04-24T14:34:41.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:41.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020025+0000", 2024-04-24T14:34:41.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9484999999999998e-05 2024-04-24T14:34:41.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:41.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020035+0000", 2024-04-24T14:34:41.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0444e-05 2024-04-24T14:34:41.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:41.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029150+0000", 2024-04-24T14:34:41.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091150160000000001 2024-04-24T14:34:41.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:41.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029352+0000", 2024-04-24T14:34:41.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020166499999999999 2024-04-24T14:34:41.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:41.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029611+0000", 2024-04-24T14:34:41.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025949600000000001 2024-04-24T14:34:41.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:41.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030518+0000", 2024-04-24T14:34:41.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090633300000000005 2024-04-24T14:34:41.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032088+0000", 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015703290000000001 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032130+0000", 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2273e-05 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032159+0000", 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8387000000000001e-05 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.339 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.340 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.340 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:41.340 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:41.340 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:41.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:34:41.341 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:41.345+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:41.342 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:41.442 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:41.445+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:41.453 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:41.453 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:41.453 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:41.453 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:41.462 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:41.510 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:41.529 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:41.542 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:41.546+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:41.549 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:41.564 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:41.564 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:41.564 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:41.564 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:41.564 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:41.572 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:41.573 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:41.594 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:41.614 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:41.617 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:41.643 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:41.646+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:41.670 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:41.670 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:41.679 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:41.743 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:41.746+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:41.777 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:41.777 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:41.786 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:41.844 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:41.849+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:41.880 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:41.880 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:41.889 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:41.944 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:41.949+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:42.030 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:42.030 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:42.039 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:42.045 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:42.049+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:42.137 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:42.137 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:42.146 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:42.149+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:42.147 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:42.205 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-24T14:34:42.262 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:42.266+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:42.298 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:42.298 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:42.308 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:42.346 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:42.353+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:42.423 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:42.423 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:42.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:42.448 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:42.450+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:42.546 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:42.546 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:42.549 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:42.553+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:42.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:42.650 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:42.653+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:42.692 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:42.692 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:42.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:42.751 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:42.754+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:42.803 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:42.803 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:42.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:42.854 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:42.858+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:42.954 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:42.957+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:42.954 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:42.954 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:42.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:43.055 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:43.058+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:43.059 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:43.059 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:43.069 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:43.156 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:43.161+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:43.173 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:43.173 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:43.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:43.257 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:43.261+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:43.277 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:43.277 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:43.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:43.357 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:43.361+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:43.387 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:43.387 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:43.403 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:43.458 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:43.462+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:43.510 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:43.510 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:43.520 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:43.559 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:43.562+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:43.624 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:43.624 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:43.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:34:43.633 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:43.674 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:43.678+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:43.730 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:43.730 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:43.739 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:43.761 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:43.765+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:43.771 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:43.794 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:43.816 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:43.838 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:43.842 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:43.842 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:43.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:43.842 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:43.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.842 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:43.842 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:43.842 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 141.21965571499999, 2024-04-24T14:34:43.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:34:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:34:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:34:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:34:43.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:34:43.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:43.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:34:43.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:34:43.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:43.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:34:43.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:34:43.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:43.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:34:43.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:34:43.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:43.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:34:43.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:34:43.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 141.21947862799999, 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:43.848 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:34:43.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:43.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:34:43.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:34:43.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:34:43.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:34:43.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:43.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 141.219374714, 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:34:43.853 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.854 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:43.854 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.854 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:43.854 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:43.854 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:34:43.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.854 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:43.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:43.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:34:43.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:34:43.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:43.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:34:43.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:34:43.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:34:43.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:34:43.858 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.858 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 141.21898626300001, 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:34:43.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:34:43.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:34:43.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:34:43.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:43.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:34:43.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:34:43.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:34:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 141.21890105099999, 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.864 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:43.865 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.865 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:43.865 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:43.865 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:43.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.865 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:43.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:43.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.870 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:43.870 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:43.870 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 141.21866417000001, 2024-04-24T14:34:43.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:34:43.870 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.870 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:43.870 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.870 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:43.870 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:43.870 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:34:43.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.871 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:43.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:43.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:34:43.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:43.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:34:43.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:34:43.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:34:43.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:34:43.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:34:43.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:34:43.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:34:43.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:34:43.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:34:43.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:34:43.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:43.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:34:43.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:34:43.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:43.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:34:43.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:34:43.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:43.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:34:43.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.876 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:43.876 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:43.876 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 141.21858264799999, 2024-04-24T14:34:43.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:34:43.876 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.876 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:43.876 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.876 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:43.876 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:43.876 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:34:43.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.876 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:34:43.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:34:43.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:34:43.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:34:43.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:34:43.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:34:43.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 140.78667030099999, 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.881 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:43.882 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:43.882 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:34:43.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.882 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:43.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:43.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:34:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:34:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:34:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:34:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.886 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 140.78664200099999, 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:34:43.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:34:43.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:34:43.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:34:43.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:43.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:34:43.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:34:43.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:43.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:34:43.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:34:43.892 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.892 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.892 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.892 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:43.892 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:43.892 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 140.78655429700001, 2024-04-24T14:34:43.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:34:43.892 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.892 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:43.892 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.892 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:34:43.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:43.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:34:43.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:34:43.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:34:43.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:34:43.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:34:43.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:43.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:34:43.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:34:43.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:43.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:34:43.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:34:43.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:43.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.898 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:43.898 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:43.898 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 140.78643149600001, 2024-04-24T14:34:43.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:34:43.898 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.898 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:43.898 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.898 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:43.898 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:43.898 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:34:43.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.898 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:34:43.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:34:43.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:34:43.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:34:43.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:34:43.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:34:43.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:34:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:43.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:34:43.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:34:43.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:43.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:34:43.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:34:43.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:43.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 71.480186935999996, 2024-04-24T14:34:43.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:34:43.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:34:43.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:34:43.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:34:43.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:34:43.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:34:43.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:34:43.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:34:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:34:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:34:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:34:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:34:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:34:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 63.786062936999997, 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.908 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:43.913 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:43.914 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 44.826702971000003, 2024-04-24T14:34:43.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:34:43.914 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.914 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:43.914 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.914 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:43.914 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:43.914 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:34:43.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.914 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:43.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:34:43.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:34:43.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:34:43.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:34:43.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:34:43.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:34:43.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:34:43.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 44.826622415999999, 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:34:43.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:34:43.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:34:43.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:34:43.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:34:43.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:43.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:34:43.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:34:43.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:34:43.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:34:43.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 44.826574686999997, 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.924 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:34:43.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:43.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:34:43.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:34:43.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:34:43.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:34:43.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:43.929 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:43.930 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 44.826546923999999, 2024-04-24T14:34:43.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:34:43.930 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.930 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:43.930 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.930 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:43.930 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:43.930 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:34:43.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.930 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:43.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:34:43.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:34:43.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:34:43.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:34:43.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:34:43.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:34:43.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:34:43.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:43.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:34:43.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:34:43.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:34:43.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:34:43.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:43.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:34:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:34:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:34:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:34:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:34:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:34:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 44.826507489000001, 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:43.935 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.936 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:43.936 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:43.936 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:34:43.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.936 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:43.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:43.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:34:43.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:43.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:34:43.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:34:43.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:34:43.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:34:43.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:34:43.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:34:43.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:34:43.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:34:43.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:34:43.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:34:43.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:34:43.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:34:43.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:34:43.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:34:43.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 44.826482505000001, 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:43.941 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:34:43.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.942 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:43.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:43.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:34:43.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:34:43.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 44.826456845999999, 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:34:43.946 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:43.947 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:43.947 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:43.947 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:43.947 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:43.947 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:34:43.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.947 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:43.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:43.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:43.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:43.952 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:43.952 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:43.953 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:43.953 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:43.954 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:43.954 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:43.954 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:43.954 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:43.954 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:43.955 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:43.955 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:43.955 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:43.954+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:44.054 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:44.057+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:44.105 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:44.105 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:44.105 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:44.105 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:44.114 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:44.159 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:44.165+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:44.209 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:44.209 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:44.209 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:44.209 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:44.209 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:44.217 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T14:34:44.219 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:44.256 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:44.261+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:44.318 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:44.318 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:44.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:44.318 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:44.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 62.657415888999999, 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:34:44.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:34:44.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.322 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 62.654381362999999, 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 62.654357177000001, 2024-04-24T14:34:44.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:34:44.327 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.327 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:44.327 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.327 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:44.327 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:44.327 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:34:44.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.327 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:44.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:44.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:44.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:44.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:34:44.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:34:44.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:34:44.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:34:44.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:34:44.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:34:44.330 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.330 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.330 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.331 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:44.331 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:44.331 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 62.654339792999998, 2024-04-24T14:34:44.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:34:44.331 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.331 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:44.331 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.331 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:44.331 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:44.331 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:34:44.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.331 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:44.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:44.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:34:44.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:44.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:34:44.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:34:44.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:34:44.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:34:44.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:34:44.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:34:44.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:34:44.335 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.335 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.335 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.335 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:44.335 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:44.335 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 62.639903701000001, 2024-04-24T14:34:44.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:34:44.335 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.335 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:44.335 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.335 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:44.335 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:44.336 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:34:44.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.336 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:44.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:44.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:34:44.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:34:44.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:34:44.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:34:44.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:34:44.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:34:44.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:34:44.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:34:44.339 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.339 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.339 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.339 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:44.339 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:44.340 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 62.639871401999997, 2024-04-24T14:34:44.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:34:44.340 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.340 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:44.340 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.340 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:44.340 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:44.340 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:34:44.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.340 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:44.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:34:44.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:34:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:34:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:34:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:34:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.344 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:44.344 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:44.344 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 62.625619125, 2024-04-24T14:34:44.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:34:44.344 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.344 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:44.344 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.344 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:44.344 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:44.344 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:34:44.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.344 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:44.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:44.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:34:44.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:44.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:34:44.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:34:44.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:34:44.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:34:44.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:34:44.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:34:44.348 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.348 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.348 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.348 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:44.348 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:44.348 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 62.625590025000001, 2024-04-24T14:34:44.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:34:44.348 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.348 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:44.348 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.348 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:44.349 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:44.349 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:34:44.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.349 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:44.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:44.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:44.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:44.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:34:44.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:34:44.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:34:44.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:34:44.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:34:44.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:34:44.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:34:44.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:34:44.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:44.352 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 62.625564095000001, 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:44.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:44.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:44.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:34:44.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:34:44.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:34:44.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:34:44.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:34:44.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:34:44.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:34:44.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:34:44.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:44.356 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 62.609843630999997, 2024-04-24T14:34:44.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:34:44.357 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.357 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:44.357 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.357 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:44.357 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:44.357 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:34:44.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.357 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:44.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:44.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:34:44.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:44.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:34:44.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:34:44.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 62.609811698999998, 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:44.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:44.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:34:44.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:44.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:34:44.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:34:44.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:34:44.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:34:44.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:34:44.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:34:44.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.364 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:44.365 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:44.365 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 62.609744182999997, 2024-04-24T14:34:44.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:34:44.365 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.365 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:44.365 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.365 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:44.365 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:44.365 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:34:44.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.365 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:44.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:44.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:34:44.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:44.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:34:44.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:34:44.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:34:44.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:34:44.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:34:44.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 62.552331299999999, 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:34:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:34:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:34:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:34:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:34:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:34:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:34:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:34:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:34:44.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:34:44.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:34:44.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:34:44.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:34:44.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:34:44.373 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.373 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.373 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.373 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:103 3.1 e87/74 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head v 87'261, mlcod=87'258)", 2024-04-24T14:34:44.373 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:44.373 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.361050096, 2024-04-24T14:34:44.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019745728000000001, 2024-04-24T14:34:44.373 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.373 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.373 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:44.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:44.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960679+0000", 2024-04-24T14:34:44.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:44.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960692+0000", 2024-04-24T14:34:44.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3709999999999999e-05 2024-04-24T14:34:44.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960722+0000", 2024-04-24T14:34:44.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9791999999999999e-05 2024-04-24T14:34:44.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960733+0000", 2024-04-24T14:34:44.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.128e-05 2024-04-24T14:34:44.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960857+0000", 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000123981 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960926+0000", 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8881999999999995e-05 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980353+0000", 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019426853000000001 2024-04-24T14:34:44.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980426+0000", 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2803000000000001e-05 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:105 3.1 e87/74 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head v 87'263, mlcod=87'258)", 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.351923562, 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014864743999999999, 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.377 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.378 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:44.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:44.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969805+0000", 2024-04-24T14:34:44.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:44.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969820+0000", 2024-04-24T14:34:44.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5116999999999999e-05 2024-04-24T14:34:44.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969852+0000", 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2135999999999999e-05 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969864+0000", 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.189e-05 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972122+0000", 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022580389999999999 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972165+0000", 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2429000000000003e-05 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984619+0000", 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012454735 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984672+0000", 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2077999999999998e-05 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.382 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:112 3.1 e87/74 3:a983798d:::rbd_data.12ff64619e2d.0000000000000006:head v 87'265, mlcod=87'258)", 2024-04-24T14:34:44.382 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:44.382 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.341584726000001, 2024-04-24T14:34:44.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012821978, 2024-04-24T14:34:44.382 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.382 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.382 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:44.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980144+0000", 2024-04-24T14:34:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980159+0000", 2024-04-24T14:34:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4769999999999999e-05 2024-04-24T14:34:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980189+0000", 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0419000000000001e-05 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980213+0000", 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4036000000000002e-05 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980452+0000", 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00023921500000000001 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980468+0000", 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5843999999999999e-05 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992932+0000", 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012463314 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992968+0000", 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6075999999999998e-05 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.385 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.386 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.386 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:113 3.1 e87/74 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head v 87'267, mlcod=87'258)", 2024-04-24T14:34:44.386 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:44.386 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.338050001999999, 2024-04-24T14:34:44.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010561022999999999, 2024-04-24T14:34:44.386 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.386 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.386 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:44.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:44.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983679+0000", 2024-04-24T14:34:44.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:44.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983699+0000", 2024-04-24T14:34:44.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0392000000000001e-05 2024-04-24T14:34:44.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983737+0000", 2024-04-24T14:34:44.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8340000000000002e-05 2024-04-24T14:34:44.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983756+0000", 2024-04-24T14:34:44.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8402000000000001e-05 2024-04-24T14:34:44.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984706+0000", 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095024400000000005 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984741+0000", 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4721000000000002e-05 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994211+0000", 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094708559999999997 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994241+0000", 2024-04-24T14:34:44.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9941e-05 2024-04-24T14:34:44.390 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.390 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.390 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.390 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:115 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:44.390 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:44.390 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.337595964999998, 2024-04-24T14:34:44.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010347242, 2024-04-24T14:34:44.390 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:44.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984132+0000", 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984142+0000", 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7370000000000001e-06 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984154+0000", 2024-04-24T14:34:44.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2288e-05 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984163+0000", 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3749999999999992e-06 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984246+0000", 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2962999999999998e-05 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984473+0000", 2024-04-24T14:34:44.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00022696399999999999 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984771+0000", 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00029739499999999999 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992998+0000", 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082275750000000009 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994299+0000", 2024-04-24T14:34:44.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013005289999999999 2024-04-24T14:34:44.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994403+0000", 2024-04-24T14:34:44.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010376499999999999 2024-04-24T14:34:44.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994482+0000", 2024-04-24T14:34:44.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9013999999999999e-05 2024-04-24T14:34:44.395 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.395 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.395 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.396 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:128 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:44.396 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:44.396 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.316405036999999, 2024-04-24T14:34:44.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011784144, 2024-04-24T14:34:44.396 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.396 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.396 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.396 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:34:44.396 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:44.396 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 128 2024-04-24T14:34:44.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.396 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005324+0000", 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005332+0000", 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2989999999999997e-06 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005354+0000", 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1705999999999999e-05 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005362+0000", 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5599999999999996e-06 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005418+0000", 2024-04-24T14:34:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6685999999999999e-05 2024-04-24T14:34:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005535+0000", 2024-04-24T14:34:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117008 2024-04-24T14:34:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005732+0000", 2024-04-24T14:34:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019718700000000001 2024-04-24T14:34:44.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:44.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016440+0000", 2024-04-24T14:34:44.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010707787999999999 2024-04-24T14:34:44.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:44.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016913+0000", 2024-04-24T14:34:44.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047270400000000001 2024-04-24T14:34:44.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017047+0000", 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013434699999999999 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017109+0000", 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2192000000000004e-05 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:130 3.1 e87/74 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 87'269, mlcod=87'265)", 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.315952277000001, 2024-04-24T14:34:44.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013945631999999999, 2024-04-24T14:34:44.402 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.402 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.402 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:44.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:44.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005777+0000", 2024-04-24T14:34:44.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:44.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005791+0000", 2024-04-24T14:34:44.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3743999999999999e-05 2024-04-24T14:34:44.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005803+0000", 2024-04-24T14:34:44.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2330999999999999e-05 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005810+0000", 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8990000000000004e-06 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005888+0000", 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7872000000000003e-05 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005911+0000", 2024-04-24T14:34:44.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3156e-05 2024-04-24T14:34:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019642+0000", 2024-04-24T14:34:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013730456 2024-04-24T14:34:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019724+0000", 2024-04-24T14:34:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1981e-05 2024-04-24T14:34:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.405 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.406 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.406 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:44.407 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:44.408 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:44.413+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:44.508 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:44.510+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:44.559 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:44.559 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:44.559 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:44.559 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:44.569 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:44.609 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:44.613+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:44.670 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:44.670 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:44.670 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:44.670 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:44.670 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:44.679 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:44.723 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:44.729+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:44.782 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:44.782 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:44.782 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:44.783 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:44.783 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.783 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:34:44.783 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:44.783 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.714315622000001, 2024-04-24T14:34:44.783 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:34:44.783 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.783 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.783 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.783 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.783 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.783 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:44.783 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.784 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.784 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.784 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.784 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:44.784 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.784 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.784 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.784 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.784 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:34:44.784 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:44.784 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.784 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.784 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.784 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:34:44.785 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:34:44.785 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.785 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.785 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.785 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:34:44.785 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:34:44.785 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.785 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.785 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.785 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:34:44.785 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:34:44.785 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.785 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.785 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.786 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:34:44.786 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:34:44.786 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.786 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.786 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.786 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:34:44.786 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:34:44.786 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.786 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.786 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.786 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:34:44.786 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:34:44.786 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.787 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.787 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.787 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:34:44.787 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:34:44.787 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.787 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.787 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.787 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.787 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.787 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:34:44.787 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:44.787 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.719501043000001, 2024-04-24T14:34:44.787 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:34:44.787 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.788 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.788 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.788 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.788 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.788 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:44.788 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.788 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.788 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.788 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.788 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:44.788 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.788 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.788 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.789 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.789 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:34:44.789 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:44.789 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.789 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.789 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.789 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:34:44.789 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:34:44.789 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.789 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.789 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.789 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:34:44.789 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:34:44.789 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.790 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.790 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.790 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:34:44.790 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:34:44.790 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.790 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.790 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.790 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:34:44.790 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:34:44.791 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:34:44.792 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.792 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.792 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.792 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.792 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.792 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:34:44.792 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:44.792 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.708571784999997, 2024-04-24T14:34:44.792 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:34:44.792 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.792 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.792 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.792 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.793 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.793 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:44.793 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.793 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.793 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.793 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.793 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:44.793 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.793 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.793 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.793 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.793 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:34:44.793 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:44.793 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.794 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.794 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.794 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:34:44.794 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:34:44.794 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.794 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.794 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.794 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:34:44.794 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:34:44.794 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.794 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.794 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.794 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:34:44.795 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:34:44.795 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.795 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.795 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.795 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:34:44.795 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:34:44.795 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.795 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.795 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.795 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:34:44.795 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:34:44.795 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.795 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.796 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.796 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:34:44.796 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:34:44.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.796 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.796 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.796 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:34:44.796 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:34:44.796 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.796 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.796 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.796 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.797 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:34:44.797 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:44.797 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.697615704, 2024-04-24T14:34:44.797 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:34:44.797 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.797 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.797 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.797 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.797 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.797 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:44.798 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.798 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.798 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.798 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:44.798 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.798 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.798 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.798 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:34:44.798 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:44.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.799 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.799 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.799 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:34:44.799 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:34:44.799 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.799 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.799 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.799 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:34:44.799 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:34:44.799 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.800 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.800 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.800 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:34:44.800 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:34:44.800 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.800 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.800 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.800 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:34:44.800 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:34:44.800 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.800 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.800 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.800 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:34:44.800 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:34:44.801 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.801 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.801 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.801 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:34:44.801 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:34:44.801 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.801 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.801 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.801 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:34:44.801 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:34:44.801 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.801 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.801 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.802 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.802 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.802 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:34:44.802 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:44.802 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.686046339000001, 2024-04-24T14:34:44.802 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:34:44.802 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.802 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.802 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.802 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.802 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.802 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:44.802 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.802 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.803 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.803 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.803 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:44.803 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.803 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.803 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.803 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.803 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:34:44.803 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:44.803 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.803 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.803 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.804 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:34:44.804 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:34:44.804 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.804 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.804 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.804 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:34:44.804 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:34:44.804 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.804 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.804 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.804 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:34:44.804 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:34:44.804 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:34:44.805 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.806 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.806 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.806 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:34:44.806 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:34:44.806 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.806 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.806 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.806 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.806 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.806 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:34:44.806 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:44.806 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.432926680000001, 2024-04-24T14:34:44.806 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:34:44.807 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.807 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.807 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.807 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.807 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.807 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:44.807 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.807 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.807 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.807 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:44.807 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.807 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.808 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.808 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:34:44.808 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:44.808 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.808 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.808 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.808 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:34:44.808 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:34:44.808 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.809 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.809 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.809 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:34:44.809 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:34:44.809 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.809 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.809 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.809 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:34:44.809 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:34:44.809 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.809 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.809 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.809 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:34:44.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:34:44.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.810 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:34:44.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:34:44.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.810 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:34:44.810 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:34:44.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.810 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.810 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:34:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:34:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:34:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.422181512000002, 2024-04-24T14:34:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:34:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.811 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.812 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.812 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:44.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.812 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:44.812 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.812 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.813 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:34:44.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:44.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.813 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.813 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:34:44.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:34:44.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.813 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.813 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:34:44.813 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:34:44.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.814 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.814 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:34:44.814 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:34:44.814 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.814 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.814 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.814 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:34:44.814 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:34:44.814 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.814 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.814 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.814 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:34:44.814 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:34:44.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.815 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.815 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.815 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:34:44.815 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:34:44.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.815 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.815 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.815 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:34:44.815 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:34:44.815 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.815 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.815 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.816 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.816 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:34:44.816 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:44.816 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.409815236, 2024-04-24T14:34:44.816 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:34:44.816 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.816 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.816 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.816 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.816 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.816 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:44.817 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.817 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.817 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:44.817 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.817 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.817 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:34:44.817 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:44.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.817 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.817 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.818 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:34:44.818 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:34:44.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.818 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.818 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.818 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:34:44.818 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:34:44.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.819 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.819 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.819 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:34:44.819 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:34:44.819 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.819 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.819 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.819 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:34:44.819 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:34:44.819 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.819 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:34:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:34:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:34:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:34:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:34:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:34:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.284879115000003, 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:44.821 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.822 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.822 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:44.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.822 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.822 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:34:44.822 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:44.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.822 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:34:44.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:34:44.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:34:44.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:34:44.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.823 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.823 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.823 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:34:44.823 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:34:44.823 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:34:44.824 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:34:44.824 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:34:44.824 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:34:44.824 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.824 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.824 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.824 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:34:44.825 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:34:44.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.825 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.825 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:34:44.825 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:34:44.825 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.825 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.825 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.825 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.825 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:34:44.825 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:44.825 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.265491271999998, 2024-04-24T14:34:44.826 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:34:44.826 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.826 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.826 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.826 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.826 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.826 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:44.826 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.826 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.826 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.826 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.826 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:44.826 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.827 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:34:44.827 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:44.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.827 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:34:44.827 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:34:44.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.827 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.828 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:34:44.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:34:44.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.828 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.828 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:34:44.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:34:44.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.828 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.828 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.828 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:34:44.828 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:34:44.828 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:34:44.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:34:44.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.829 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.829 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:34:44.829 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:34:44.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.829 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.830 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.830 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:34:44.830 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:34:44.830 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.830 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.830 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.830 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.830 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:34:44.830 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:44.830 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.255763428999998, 2024-04-24T14:34:44.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:34:44.831 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.831 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.831 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.831 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.831 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:44.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.831 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.831 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.831 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:44.831 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:34:44.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:44.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:34:44.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:34:44.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.832 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.832 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:34:44.832 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:34:44.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.833 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.833 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:34:44.833 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:34:44.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.833 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.833 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:34:44.833 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:34:44.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.833 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.833 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.833 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:34:44.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:34:44.834 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.834 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.834 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:34:44.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:34:44.834 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.834 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.834 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.834 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:34:44.834 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:34:44.834 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.835 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.835 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.835 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:34:44.835 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:44.835 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.095653749, 2024-04-24T14:34:44.835 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:34:44.835 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.835 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.835 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.835 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.835 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:44.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:44.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.836 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.836 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.836 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:34:44.836 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:44.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:34:44.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:34:44.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:34:44.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:34:44.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.837 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.837 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.837 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:34:44.837 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:34:44.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:34:44.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:34:44.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:34:44.838 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:34:44.838 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.838 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.838 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.838 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:34:44.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:34:44.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.839 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.839 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.839 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:34:44.839 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:34:44.839 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.839 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.840 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.840 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:34:44.840 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:44.840 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.085924753, 2024-04-24T14:34:44.840 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:34:44.840 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.840 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.840 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.840 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.840 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.840 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:44.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:44.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.841 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:34:44.841 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:44.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.841 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.841 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:34:44.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:34:44.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:34:44.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:34:44.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.842 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.842 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.842 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:34:44.842 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:34:44.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:34:44.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:34:44.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:34:44.843 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:34:44.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.843 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.843 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.843 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:34:44.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:34:44.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.844 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.844 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.844 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:34:44.844 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:34:44.844 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.844 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.844 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.844 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 45.750953918, 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.845 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:44.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:44.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:44.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.846 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.846 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.846 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:34:44.846 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:34:44.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:34:44.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:34:44.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:34:44.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:44.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:34:44.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:34:44.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:34:44.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:34:44.849 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.849 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.849 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.849 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:44.849 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:44.849 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.906803170000003, 2024-04-24T14:34:44.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:34:44.849 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.849 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:44.850 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:44.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:44.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:44.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:44.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:34:44.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:34:44.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:34:44.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:34:44.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:34:44.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:34:44.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:34:44.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:34:44.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:34:44.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:34:44.852 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.852 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.853 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.853 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:5 3.2 3.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:34:44.853 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:44.853 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.578565527999999, 2024-04-24T14:34:44.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043717790000000001, 2024-04-24T14:34:44.853 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.853 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-24T14:34:44.853 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:44.853 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:34:44.853 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:34:44.853 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 5 2024-04-24T14:34:44.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207678+0000", 2024-04-24T14:34:44.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999962 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207700+0000", 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1313000000000001e-05 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207757+0000", 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7499e-05 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207779+0000", 2024-04-24T14:34:44.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1478999999999999e-05 2024-04-24T14:34:44.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.212054+0000", 2024-04-24T14:34:44.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042750849999999996 2024-04-24T14:34:44.856 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.856 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.856 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.856 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:44.856 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:44.856 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.683040169999998, 2024-04-24T14:34:44.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044084859999999997, 2024-04-24T14:34:44.857 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.857 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:44.857 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:44.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:44.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103206+0000", 2024-04-24T14:34:44.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:44.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104731+0000", 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001524755 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104759+0000", 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7973000000000001e-05 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104807+0000", 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8112000000000003e-05 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104907+0000", 2024-04-24T14:34:44.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100317 2024-04-24T14:34:44.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104918+0000", 2024-04-24T14:34:44.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747999999999999e-05 2024-04-24T14:34:44.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.860 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.860 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.107616+0000", 2024-04-24T14:34:44.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026977559999999999 2024-04-24T14:34:44.860 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.861 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.861 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.861 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(3.2 83/82 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:44.861 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:44.861 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.676645792999999, 2024-04-24T14:34:44.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041425760000000002, 2024-04-24T14:34:44.861 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.861 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:44.861 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:44.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:44.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109600+0000", 2024-04-24T14:34:44.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:44.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111210+0000", 2024-04-24T14:34:44.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016097279999999999 2024-04-24T14:34:44.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111237+0000", 2024-04-24T14:34:44.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7189999999999999e-05 2024-04-24T14:34:44.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111251+0000", 2024-04-24T14:34:44.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.365e-05 2024-04-24T14:34:44.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111367+0000", 2024-04-24T14:34:44.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000115703 2024-04-24T14:34:44.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.864 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.864 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111378+0000", 2024-04-24T14:34:44.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1873e-05 2024-04-24T14:34:44.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.113744+0000", 2024-04-24T14:34:44.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023657069999999999 2024-04-24T14:34:44.865 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.865 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.865 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.865 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:44.866 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:44.866 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.659745217000001, 2024-04-24T14:34:44.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046193859999999996, 2024-04-24T14:34:44.866 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.866 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:44.866 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:44.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:44.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126501+0000", 2024-04-24T14:34:44.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:44.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128169+0000", 2024-04-24T14:34:44.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016686229999999999 2024-04-24T14:34:44.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128202+0000", 2024-04-24T14:34:44.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2539000000000001e-05 2024-04-24T14:34:44.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128222+0000", 2024-04-24T14:34:44.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0149e-05 2024-04-24T14:34:44.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128375+0000", 2024-04-24T14:34:44.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000153459 2024-04-24T14:34:44.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128399+0000", 2024-04-24T14:34:44.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3393000000000001e-05 2024-04-24T14:34:44.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.131122+0000", 2024-04-24T14:34:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027227140000000002 2024-04-24T14:34:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 49'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@49'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 26.637963837000001, 2024-04-24T14:34:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038414529999999999, 2024-04-24T14:34:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:44.870 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148282+0000", 2024-04-24T14:34:44.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:44.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:44.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149875+0000", 2024-04-24T14:34:44.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015929900000000001 2024-04-24T14:34:44.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:44.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149908+0000", 2024-04-24T14:34:44.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3102000000000003e-05 2024-04-24T14:34:44.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:44.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149941+0000", 2024-04-24T14:34:44.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2480000000000001e-05 2024-04-24T14:34:44.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:44.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150031+0000", 2024-04-24T14:34:44.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0394000000000002e-05 2024-04-24T14:34:44.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:44.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150043+0000", 2024-04-24T14:34:44.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1967000000000001e-05 2024-04-24T14:34:44.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:44.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:44.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:44.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.152125+0000", 2024-04-24T14:34:44.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020817029999999999 2024-04-24T14:34:44.874 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.874 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.874 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.874 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:44.874 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:44.874 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:44.876 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-24T14:34:44.877 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:44.877 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:44.878+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:44.977 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:44.980+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:44.987 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:44.987 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:44.987 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:44.987 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:44.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:45.078 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:45.080+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:45.098 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:45.099 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:45.099 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:45.099 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:45.099 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:45.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:45.178 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:45.180+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:45.216 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:45.216 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:45.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:45.217 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:45.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.217 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:45.217 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:45.217 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.340203504999998, 2024-04-24T14:34:45.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:34:45.217 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.217 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.217 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.217 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:45.217 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:45.217 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:34:45.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.218 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:45.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:45.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:34:45.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:45.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:34:45.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:34:45.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:34:45.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:34:45.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:34:45.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:34:45.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:34:45.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:34:45.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:45.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:34:45.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:34:45.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:34:45.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:34:45.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:45.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:34:45.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:34:45.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:34:45.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:34:45.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:34:45.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:34:45.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:34:45.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:34:45.223 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.223 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.223 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.223 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:45.223 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.340174519999998, 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:45.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:45.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:34:45.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:34:45.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:34:45.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:34:45.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:34:45.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:34:45.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:34:45.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:34:45.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:45.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:34:45.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:34:45.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:34:45.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:34:45.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:45.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:34:45.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:34:45.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:34:45.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:34:45.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:34:45.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:34:45.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:34:45.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:34:45.229 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.229 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.229 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.229 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:45.229 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:45.229 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.340156035, 2024-04-24T14:34:45.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:34:45.230 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.230 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.230 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.230 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:45.230 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:45.230 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:34:45.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.230 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:45.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:45.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:34:45.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:34:45.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:34:45.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:34:45.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:34:45.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:34:45.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:34:45.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:34:45.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:34:45.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:45.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:34:45.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:34:45.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:34:45.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:34:45.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:45.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:34:45.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:34:45.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:34:45.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:34:45.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:34:45.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:34:45.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:34:45.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:34:45.235 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.235 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.235 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.235 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:45.235 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:45.236 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.340140753999997, 2024-04-24T14:34:45.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:34:45.236 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.236 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.236 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.236 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:45.236 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:45.236 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:34:45.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.236 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:45.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:45.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:45.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:34:45.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:34:45.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:34:45.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:34:45.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:34:45.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:34:45.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:34:45.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:34:45.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:45.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:34:45.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:34:45.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:34:45.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:34:45.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:45.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:34:45.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:34:45.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:34:45.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:34:45.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:34:45.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:34:45.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:34:45.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:34:45.242 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.242 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.242 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.242 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:45.242 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:45.242 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.340124973000002, 2024-04-24T14:34:45.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:34:45.242 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.242 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.242 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:45.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:45.243 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:34:45.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.243 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:45.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:45.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:45.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:34:45.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:34:45.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:34:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:34:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:34:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:34:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:34:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:34:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:45.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:34:45.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:34:45.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:34:45.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:34:45.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:45.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:34:45.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:34:45.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:34:45.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:34:45.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:34:45.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:34:45.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:34:45.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:34:45.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.248 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.248 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:45.248 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:45.248 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.340110684999999, 2024-04-24T14:34:45.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:34:45.248 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.249 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:45.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:45.249 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:34:45.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.249 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:45.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:45.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:45.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:34:45.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:34:45.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:34:45.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:34:45.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:34:45.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:34:45.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:34:45.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:34:45.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:45.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:34:45.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:34:45.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:34:45.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:34:45.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:45.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:34:45.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:34:45.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:34:45.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:34:45.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:34:45.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:34:45.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:34:45.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:34:45.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.255 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.255 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:45.255 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:45.255 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.340090932999999, 2024-04-24T14:34:45.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:34:45.256 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.256 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:45.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:45.256 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:34:45.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.256 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:45.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:45.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:45.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:34:45.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:34:45.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:34:45.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:34:45.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:34:45.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:34:45.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:34:45.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:34:45.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:45.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:34:45.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:34:45.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:34:45.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:34:45.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:45.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:34:45.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:34:45.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:34:45.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:34:45.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:34:45.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:34:45.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:34:45.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:34:45.261 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.262 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.262 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.262 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:45.262 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:45.262 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.340073486000001, 2024-04-24T14:34:45.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:34:45.262 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.262 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.262 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.262 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:45.263 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:45.263 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:34:45.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.263 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:45.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:45.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:45.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:34:45.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:34:45.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:34:45.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:34:45.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:34:45.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:34:45.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:34:45.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:34:45.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:45.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:34:45.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:34:45.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:34:45.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:34:45.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:45.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:34:45.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:34:45.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:34:45.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:34:45.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:34:45.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:34:45.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:34:45.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:34:45.268 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.268 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.268 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.268 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:100 3.2 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.268 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:45.269 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.466018671, 2024-04-24T14:34:45.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009078015, 2024-04-24T14:34:45.269 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.269 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.269 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.269 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.269 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.269 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 100 2024-04-24T14:34:45.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.269 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:45.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:45.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754019+0000", 2024-04-24T14:34:45.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754025+0000", 2024-04-24T14:34:45.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2110000000000003e-06 2024-04-24T14:34:45.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754037+0000", 2024-04-24T14:34:45.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1868999999999999e-05 2024-04-24T14:34:45.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754046+0000", 2024-04-24T14:34:45.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7430000000000008e-06 2024-04-24T14:34:45.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754095+0000", 2024-04-24T14:34:45.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8924000000000001e-05 2024-04-24T14:34:45.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754153+0000", 2024-04-24T14:34:45.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8165000000000003e-05 2024-04-24T14:34:45.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:45.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754262+0000", 2024-04-24T14:34:45.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108694 2024-04-24T14:34:45.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.762999+0000", 2024-04-24T14:34:45.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087372590000000007 2024-04-24T14:34:45.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763027+0000", 2024-04-24T14:34:45.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8467e-05 2024-04-24T14:34:45.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763070+0000", 2024-04-24T14:34:45.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2236e-05 2024-04-24T14:34:45.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763098+0000", 2024-04-24T14:34:45.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8561999999999999e-05 2024-04-24T14:34:45.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.278 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.279 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:101 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [delete] snapc 7=[7,6,4] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.279 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:45.279 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.260811005000001, 2024-04-24T14:34:45.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012894743, 2024-04-24T14:34:45.279 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.279 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 101 2024-04-24T14:34:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.280 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959227+0000", 2024-04-24T14:34:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959233+0000", 2024-04-24T14:34:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.857e-06 2024-04-24T14:34:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959249+0000", 2024-04-24T14:34:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6155999999999999e-05 2024-04-24T14:34:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959257+0000", 2024-04-24T14:34:45.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7440000000000007e-06 2024-04-24T14:34:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959790+0000", 2024-04-24T14:34:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053381100000000003 2024-04-24T14:34:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960206+0000", 2024-04-24T14:34:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041549700000000002 2024-04-24T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960744+0000", 2024-04-24T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053762199999999997 2024-04-24T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971963+0000", 2024-04-24T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011219816000000001 2024-04-24T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972021+0000", 2024-04-24T14:34:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7855000000000001e-05 2024-04-24T14:34:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972073+0000", 2024-04-24T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2196e-05 2024-04-24T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972123+0000", 2024-04-24T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9311999999999998e-05 2024-04-24T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:102 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.260770979, 2024-04-24T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012939713, 2024-04-24T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 102 2024-04-24T14:34:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959267+0000", 2024-04-24T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959274+0000", 2024-04-24T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.593e-06 2024-04-24T14:34:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959291+0000", 2024-04-24T14:34:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6387000000000001e-05 2024-04-24T14:34:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959301+0000", 2024-04-24T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0023e-05 2024-04-24T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969594+0000", 2024-04-24T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010293182 2024-04-24T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969902+0000", 2024-04-24T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030835299999999998 2024-04-24T14:34:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970408+0000", 2024-04-24T14:34:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050588100000000002 2024-04-24T14:34:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972009+0000", 2024-04-24T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016011210000000001 2024-04-24T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972132+0000", 2024-04-24T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122521 2024-04-24T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972174+0000", 2024-04-24T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2242e-05 2024-04-24T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972208+0000", 2024-04-24T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3522999999999997e-05 2024-04-24T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:104 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.260159765000001, 2024-04-24T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014589062, 2024-04-24T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 104 2024-04-24T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959877+0000", 2024-04-24T14:34:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959890+0000", 2024-04-24T14:34:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2537e-05 2024-04-24T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959909+0000", 2024-04-24T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9340000000000001e-05 2024-04-24T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959922+0000", 2024-04-24T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2882e-05 2024-04-24T14:34:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970900+0000", 2024-04-24T14:34:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010977736 2024-04-24T14:34:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971039+0000", 2024-04-24T14:34:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139593 2024-04-24T14:34:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971453+0000", 2024-04-24T14:34:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041351099999999998 2024-04-24T14:34:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974044+0000", 2024-04-24T14:34:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025908289999999998 2024-04-24T14:34:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974377+0000", 2024-04-24T14:34:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033310399999999997 2024-04-24T14:34:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974433+0000", 2024-04-24T14:34:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6166999999999999e-05 2024-04-24T14:34:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974468+0000", 2024-04-24T14:34:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5336000000000002e-05 2024-04-24T14:34:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:106 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.260100432000002, 2024-04-24T14:34:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017192064999999999, 2024-04-24T14:34:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 106 2024-04-24T14:34:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959938+0000", 2024-04-24T14:34:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959943+0000", 2024-04-24T14:34:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.677e-06 2024-04-24T14:34:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959953+0000", 2024-04-24T14:34:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8069999999999994e-06 2024-04-24T14:34:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:34:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0560000000000001e-06 2024-04-24T14:34:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972232+0000", 2024-04-24T14:34:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012272460000000001 2024-04-24T14:34:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972304+0000", 2024-04-24T14:34:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2630000000000001e-05 2024-04-24T14:34:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972548+0000", 2024-04-24T14:34:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000244047 2024-04-24T14:34:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976900+0000", 2024-04-24T14:34:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043514069999999998 2024-04-24T14:34:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977076+0000", 2024-04-24T14:34:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176281 2024-04-24T14:34:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977106+0000", 2024-04-24T14:34:45.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0015e-05 2024-04-24T14:34:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977131+0000", 2024-04-24T14:34:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4669e-05 2024-04-24T14:34:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.323 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:114 3.2 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:45.324 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.236779267999999, 2024-04-24T14:34:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090880089999999993, 2024-04-24T14:34:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 114 2024-04-24T14:34:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983259+0000", 2024-04-24T14:34:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983264+0000", 2024-04-24T14:34:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9130000000000002e-06 2024-04-24T14:34:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983277+0000", 2024-04-24T14:34:45.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3723999999999999e-05 2024-04-24T14:34:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983282+0000", 2024-04-24T14:34:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0429999999999998e-06 2024-04-24T14:34:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983339+0000", 2024-04-24T14:34:45.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6323000000000003e-05 2024-04-24T14:34:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983464+0000", 2024-04-24T14:34:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012554299999999999 2024-04-24T14:34:45.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983693+0000", 2024-04-24T14:34:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000228848 2024-04-24T14:34:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.986978+0000", 2024-04-24T14:34:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032852810000000001 2024-04-24T14:34:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992288+0000", 2024-04-24T14:34:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053095369999999996 2024-04-24T14:34:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992322+0000", 2024-04-24T14:34:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3683999999999998e-05 2024-04-24T14:34:45.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992348+0000", 2024-04-24T14:34:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6157999999999999e-05 2024-04-24T14:34:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:121 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.224723169000001, 2024-04-24T14:34:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098798159999999996, 2024-04-24T14:34:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.336 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 121 2024-04-24T14:34:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995315+0000", 2024-04-24T14:34:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995320+0000", 2024-04-24T14:34:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0239999999999997e-06 2024-04-24T14:34:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995328+0000", 2024-04-24T14:34:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7800000000000001e-06 2024-04-24T14:34:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995334+0000", 2024-04-24T14:34:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2330000000000003e-06 2024-04-24T14:34:45.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998810+0000", 2024-04-24T14:34:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034756090000000002 2024-04-24T14:34:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998870+0000", 2024-04-24T14:34:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0625999999999999e-05 2024-04-24T14:34:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.999091+0000", 2024-04-24T14:34:45.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000220631 2024-04-24T14:34:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.001389+0000", 2024-04-24T14:34:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022977549999999998 2024-04-24T14:34:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005097+0000", 2024-04-24T14:34:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037085439999999998 2024-04-24T14:34:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005154+0000", 2024-04-24T14:34:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6208000000000001e-05 2024-04-24T14:34:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005196+0000", 2024-04-24T14:34:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2052000000000003e-05 2024-04-24T14:34:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:127 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.342 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.214421652999999, 2024-04-24T14:34:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013414918, 2024-04-24T14:34:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2024-04-24T14:34:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:45.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005616+0000", 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005622+0000", 2024-04-24T14:34:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2020000000000003e-06 2024-04-24T14:34:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005632+0000", 2024-04-24T14:34:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2024-04-24T14:34:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005637+0000", 2024-04-24T14:34:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7439999999999998e-06 2024-04-24T14:34:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005947+0000", 2024-04-24T14:34:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000310792 2024-04-24T14:34:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006059+0000", 2024-04-24T14:34:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011192399999999999 2024-04-24T14:34:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006503+0000", 2024-04-24T14:34:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000443394 2024-04-24T14:34:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018864+0000", 2024-04-24T14:34:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012361162 2024-04-24T14:34:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018877+0000", 2024-04-24T14:34:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3497e-05 2024-04-24T14:34:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018965+0000", 2024-04-24T14:34:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7476000000000001e-05 2024-04-24T14:34:45.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019032+0000", 2024-04-24T14:34:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7469000000000001e-05 2024-04-24T14:34:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:129 3.2 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.214281419999999, 2024-04-24T14:34:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010059754000000001, 2024-04-24T14:34:45.348 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 129 2024-04-24T14:34:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005757+0000", 2024-04-24T14:34:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005760+0000", 2024-04-24T14:34:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2430000000000001e-06 2024-04-24T14:34:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005769+0000", 2024-04-24T14:34:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4810000000000006e-06 2024-04-24T14:34:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005773+0000", 2024-04-24T14:34:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005840+0000", 2024-04-24T14:34:45.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6934999999999995e-05 2024-04-24T14:34:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005961+0000", 2024-04-24T14:34:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012153 2024-04-24T14:34:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006181+0000", 2024-04-24T14:34:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021947500000000001 2024-04-24T14:34:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015520+0000", 2024-04-24T14:34:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093393049999999991 2024-04-24T14:34:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015733+0000", 2024-04-24T14:34:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021355699999999999 2024-04-24T14:34:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015773+0000", 2024-04-24T14:34:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9554000000000001e-05 2024-04-24T14:34:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015817+0000", 2024-04-24T14:34:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4325e-05 2024-04-24T14:34:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:131 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.214003539, 2024-04-24T14:34:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014251976, 2024-04-24T14:34:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.354 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131 2024-04-24T14:34:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:45.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006034+0000", 2024-04-24T14:34:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006042+0000", 2024-04-24T14:34:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2970000000000004e-06 2024-04-24T14:34:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006061+0000", 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8505999999999999e-05 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006071+0000", 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0409e-05 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007899+0000", 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018274789999999999 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008281+0000", 2024-04-24T14:34:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038244700000000001 2024-04-24T14:34:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008528+0000", 2024-04-24T14:34:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024723900000000001 2024-04-24T14:34:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019070+0000", 2024-04-24T14:34:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010541867999999999 2024-04-24T14:34:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020099+0000", 2024-04-24T14:34:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00102852 2024-04-24T14:34:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020226+0000", 2024-04-24T14:34:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012744300000000001 2024-04-24T14:34:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020287+0000", 2024-04-24T14:34:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1253000000000005e-05 2024-04-24T14:34:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.359 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:132 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.213949685999999, 2024-04-24T14:34:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015917981000000001, 2024-04-24T14:34:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 132 2024-04-24T14:34:45.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006093+0000", 2024-04-24T14:34:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3819999999999997e-06 2024-04-24T14:34:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006103+0000", 2024-04-24T14:34:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0352e-05 2024-04-24T14:34:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006110+0000", 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3910000000000003e-06 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015586+0000", 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094759119999999995 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015730+0000", 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014487199999999999 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015886+0000", 2024-04-24T14:34:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015573799999999999 2024-04-24T14:34:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020306+0000", 2024-04-24T14:34:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044199399999999998 2024-04-24T14:34:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021887+0000", 2024-04-24T14:34:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015809400000000001 2024-04-24T14:34:45.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021942+0000", 2024-04-24T14:34:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5071000000000002e-05 2024-04-24T14:34:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.022007+0000", 2024-04-24T14:34:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5110000000000005e-05 2024-04-24T14:34:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.5324.0:127 3.0 e87/69 ondisk, result = 0)", 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.212240617999999, 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011245125, 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007804+0000", 2024-04-24T14:34:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6610000000000003e-06 2024-04-24T14:34:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007817+0000", 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3030999999999999e-05 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007832+0000", 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.47e-05 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018843+0000", 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011010647 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018853+0000", 2024-04-24T14:34:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319000000000001e-05 2024-04-24T14:34:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019043+0000", 2024-04-24T14:34:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019055600000000001 2024-04-24T14:34:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.369 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.369 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:45.371 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:45.374+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:45.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:45.472 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:45.474+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:45.524 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:45.524 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:45.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:45.578 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:45.582+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:45.631 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:45.631 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:45.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:45.673 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:45.676+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:45.744 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:45.744 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.993180584999998, 2024-04-24T14:34:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:34:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:45.745 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:34:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:34:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:34:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:34:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:34:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:34:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:34:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:34:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:34:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:34:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:34:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:34:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:34:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:34:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:34:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:34:45.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:34:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:34:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:34:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:34:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:34:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:34:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.993129750999998, 2024-04-24T14:34:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:34:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.751 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:34:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:45.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:34:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:34:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:34:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:34:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:34:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:34:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:34:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:34:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:34:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:34:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:34:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:34:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:34:45.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:34:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:34:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:34:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:34:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:34:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:34:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:34:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.993093444000003, 2024-04-24T14:34:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:34:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.757 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:34:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:34:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:34:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:34:45.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:34:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:34:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:34:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:34:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:34:45.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:34:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:34:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:34:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:34:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:34:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:34:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:34:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:34:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:34:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:34:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:34:45.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:34:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:34:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.989682193999997, 2024-04-24T14:34:45.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:34:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:34:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:34:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:34:45.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:34:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:34:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:34:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:34:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:34:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:34:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:34:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:34:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:34:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:34:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:34:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:34:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:34:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:34:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:34:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:34:45.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:34:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:34:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:34:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.989620516999999, 2024-04-24T14:34:45.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:34:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:34:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:34:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:34:45.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:34:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:34:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:34:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:34:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:34:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:34:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:34:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:34:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:34:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:34:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:34:45.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:34:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:34:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:34:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:34:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:34:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:34:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:34:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:45.775 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.989585941000001, 2024-04-24T14:34:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:34:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:34:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:34:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:34:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:34:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:34:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:34:45.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:34:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:34:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:34:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:34:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:34:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:34:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:34:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:34:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:34:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:34:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:34:45.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.781 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.989556602, 2024-04-24T14:34:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:34:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:34:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:34:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:34:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:34:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:34:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:34:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:34:45.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:34:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:34:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:34:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:34:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:34:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:34:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:34:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:34:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:34:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:34:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:34:45.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:34:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:34:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:34:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:34:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.989480020000002, 2024-04-24T14:34:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:34:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:34:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:34:45.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:34:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:34:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:34:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:34:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:34:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:34:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:34:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:34:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:34:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:34:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:34:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:34:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:34:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:34:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:34:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:34:45.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:34:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:34:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:34:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:34:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.989432475000001, 2024-04-24T14:34:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:34:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:34:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:34:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:34:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:34:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:34:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:34:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:34:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:34:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:34:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:34:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:34:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:34:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:34:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:34:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:34:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:34:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.799 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.989391832999999, 2024-04-24T14:34:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:34:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:34:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:34:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:34:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:34:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:34:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:34:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:34:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:34:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:34:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:34:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:34:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:34:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:34:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:34:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:34:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:34:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:34:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:34:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:34:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:34:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:34:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.989374722000001, 2024-04-24T14:34:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:34:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:34:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.806 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:34:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:34:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:34:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:34:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:34:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:34:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:34:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:34:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:34:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:34:45.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:34:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:34:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:34:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:34:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:34:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.812 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:103 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.812 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:45.812 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.789808609000001, 2024-04-24T14:34:45.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032646351999999997, 2024-04-24T14:34:45.812 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.812 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.812 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.812 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.812 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.812 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 103 2024-04-24T14:34:45.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.812 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:45.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:45.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959674+0000", 2024-04-24T14:34:45.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:45.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959688+0000", 2024-04-24T14:34:45.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2024-04-24T14:34:45.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959722+0000", 2024-04-24T14:34:45.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3936000000000002e-05 2024-04-24T14:34:45.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959735+0000", 2024-04-24T14:34:45.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2926e-05 2024-04-24T14:34:45.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959921+0000", 2024-04-24T14:34:45.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185601 2024-04-24T14:34:45.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960324+0000", 2024-04-24T14:34:45.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040356700000000002 2024-04-24T14:34:45.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:45.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960839+0000", 2024-04-24T14:34:45.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051410599999999996 2024-04-24T14:34:45.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.975634+0000", 2024-04-24T14:34:45.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014795770999999999 2024-04-24T14:34:45.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992113+0000", 2024-04-24T14:34:45.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016478982999999999 2024-04-24T14:34:45.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992228+0000", 2024-04-24T14:34:45.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114295 2024-04-24T14:34:45.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992323+0000", 2024-04-24T14:34:45.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5344000000000001e-05 2024-04-24T14:34:45.817 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.817 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.817 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.818 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:105 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.818 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:45.818 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.789558831000001, 2024-04-24T14:34:45.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032581757000000003, 2024-04-24T14:34:45.818 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.818 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.818 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.818 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.818 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.818 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105 2024-04-24T14:34:45.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.818 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:45.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:45.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959925+0000", 2024-04-24T14:34:45.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:45.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959937+0000", 2024-04-24T14:34:45.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2391999999999999e-05 2024-04-24T14:34:45.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:34:45.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2535999999999999e-05 2024-04-24T14:34:45.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959970+0000", 2024-04-24T14:34:45.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1039999999999999e-05 2024-04-24T14:34:45.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969531+0000", 2024-04-24T14:34:45.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095603949999999993 2024-04-24T14:34:45.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969669+0000", 2024-04-24T14:34:45.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013806499999999999 2024-04-24T14:34:45.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:45.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970010+0000", 2024-04-24T14:34:45.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034062200000000001 2024-04-24T14:34:45.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976983+0000", 2024-04-24T14:34:45.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069735600000000002 2024-04-24T14:34:45.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992413+0000", 2024-04-24T14:34:45.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015429841999999999 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992457+0000", 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4169999999999999e-05 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992508+0000", 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0995000000000002e-05 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:113 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.769332540000001, 2024-04-24T14:34:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014793863000000001, 2024-04-24T14:34:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 113 2024-04-24T14:34:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980152+0000", 2024-04-24T14:34:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980156+0000", 2024-04-24T14:34:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.619e-06 2024-04-24T14:34:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980163+0000", 2024-04-24T14:34:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7070000000000003e-06 2024-04-24T14:34:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980173+0000", 2024-04-24T14:34:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0957e-05 2024-04-24T14:34:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983381+0000", 2024-04-24T14:34:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032079930000000001 2024-04-24T14:34:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983628+0000", 2024-04-24T14:34:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024665700000000001 2024-04-24T14:34:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983964+0000", 2024-04-24T14:34:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000336332 2024-04-24T14:34:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992694+0000", 2024-04-24T14:34:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087297009999999994 2024-04-24T14:34:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994878+0000", 2024-04-24T14:34:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021841439999999998 2024-04-24T14:34:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994917+0000", 2024-04-24T14:34:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8544999999999998e-05 2024-04-24T14:34:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994947+0000", 2024-04-24T14:34:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9705000000000001e-05 2024-04-24T14:34:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:130 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.743639105, 2024-04-24T14:34:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014646289999999999, 2024-04-24T14:34:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 130 2024-04-24T14:34:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005844+0000", 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005853+0000", 2024-04-24T14:34:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1319999999999994e-06 2024-04-24T14:34:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005870+0000", 2024-04-24T14:34:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7448999999999998e-05 2024-04-24T14:34:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005876+0000", 2024-04-24T14:34:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3319999999999997e-06 2024-04-24T14:34:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005913+0000", 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6835999999999998e-05 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006017+0000", 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010358500000000001 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006139+0000", 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001227 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.010194+0000", 2024-04-24T14:34:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040549380000000001 2024-04-24T14:34:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020365+0000", 2024-04-24T14:34:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010170983 2024-04-24T14:34:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020428+0000", 2024-04-24T14:34:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2457000000000002e-05 2024-04-24T14:34:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020492+0000", 2024-04-24T14:34:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4592999999999996e-05 2024-04-24T14:34:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:155 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1601536~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.709978318000001, 2024-04-24T14:34:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015144391, 2024-04-24T14:34:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 155 2024-04-24T14:34:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:45.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039506+0000", 2024-04-24T14:34:45.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:45.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039766+0000", 2024-04-24T14:34:45.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025994400000000002 2024-04-24T14:34:45.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039780+0000", 2024-04-24T14:34:45.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3366e-05 2024-04-24T14:34:45.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039787+0000", 2024-04-24T14:34:45.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4359999999999996e-06 2024-04-24T14:34:45.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.050886+0000", 2024-04-24T14:34:45.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011098993 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051026+0000", 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001395 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051211+0000", 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018547099999999999 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054490+0000", 2024-04-24T14:34:45.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032785499999999999 2024-04-24T14:34:45.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054505+0000", 2024-04-24T14:34:45.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5906999999999999e-05 2024-04-24T14:34:45.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054594+0000", 2024-04-24T14:34:45.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8817e-05 2024-04-24T14:34:45.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054651+0000", 2024-04-24T14:34:45.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7040999999999998e-05 2024-04-24T14:34:45.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.841 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.841 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:156 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1732608~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.841 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:45.841 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.709687612, 2024-04-24T14:34:45.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016895194999999998, 2024-04-24T14:34:45.841 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.841 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.841 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.842 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.842 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.842 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 156 2024-04-24T14:34:45.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.842 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:45.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:45.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039797+0000", 2024-04-24T14:34:45.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040065+0000", 2024-04-24T14:34:45.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000268361 2024-04-24T14:34:45.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040086+0000", 2024-04-24T14:34:45.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1024000000000001e-05 2024-04-24T14:34:45.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040094+0000", 2024-04-24T14:34:45.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9379999999999992e-06 2024-04-24T14:34:45.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051936+0000", 2024-04-24T14:34:45.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011842518 2024-04-24T14:34:45.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052062+0000", 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125482 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052206+0000", 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143953 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054715+0000", 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025088240000000002 2024-04-24T14:34:45.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056534+0000", 2024-04-24T14:34:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001819436 2024-04-24T14:34:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056610+0000", 2024-04-24T14:34:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5383000000000007e-05 2024-04-24T14:34:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056693+0000", 2024-04-24T14:34:45.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3283999999999997e-05 2024-04-24T14:34:45.847 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.847 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.847 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.847 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:157 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1863680~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.847 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:45.847 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.709380233000001, 2024-04-24T14:34:45.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016763674999999999, 2024-04-24T14:34:45.847 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.847 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.847 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.848 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.848 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.848 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157 2024-04-24T14:34:45.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.848 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:45.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:45.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040104+0000", 2024-04-24T14:34:45.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040244+0000", 2024-04-24T14:34:45.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014035499999999999 2024-04-24T14:34:45.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040255+0000", 2024-04-24T14:34:45.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0135999999999999e-05 2024-04-24T14:34:45.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040260+0000", 2024-04-24T14:34:45.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7919999999999998e-06 2024-04-24T14:34:45.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053406+0000", 2024-04-24T14:34:45.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013146023999999999 2024-04-24T14:34:45.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053531+0000", 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012471800000000001 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053715+0000", 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018395099999999999 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056711+0000", 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029956520000000001 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056783+0000", 2024-04-24T14:34:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2010999999999999e-05 2024-04-24T14:34:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056831+0000", 2024-04-24T14:34:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8189999999999998e-05 2024-04-24T14:34:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056869+0000", 2024-04-24T14:34:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7812000000000003e-05 2024-04-24T14:34:45.852 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.853 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.853 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:158 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1994752~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.853 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:45.853 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.709218306, 2024-04-24T14:34:45.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016845334999999999, 2024-04-24T14:34:45.853 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.853 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:45.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.853 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2024-04-24T14:34:45.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.854 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:45.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:45.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040266+0000", 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040379+0000", 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112728 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040387+0000", 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8499999999999994e-06 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040391+0000", 2024-04-24T14:34:45.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2719999999999999e-06 2024-04-24T14:34:45.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054803+0000", 2024-04-24T14:34:45.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014411548 2024-04-24T14:34:45.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054982+0000", 2024-04-24T14:34:45.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017941699999999999 2024-04-24T14:34:45.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:45.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.055167+0000", 2024-04-24T14:34:45.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018542100000000001 2024-04-24T14:34:45.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:45.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056718+0000", 2024-04-24T14:34:45.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015505670000000001 2024-04-24T14:34:45.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:45.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057020+0000", 2024-04-24T14:34:45.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030227500000000002 2024-04-24T14:34:45.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:45.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057070+0000", 2024-04-24T14:34:45.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9611000000000003e-05 2024-04-24T14:34:45.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057112+0000", 2024-04-24T14:34:45.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2373999999999998e-05 2024-04-24T14:34:45.858 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.858 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.859 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:159 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:45.859 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:45.859 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.703783317999999, 2024-04-24T14:34:45.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015930650000000001, 2024-04-24T14:34:45.859 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:45.859 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:45.859 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:45.859 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:45.859 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:45.859 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045701+0000", 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045707+0000", 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.562e-06 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045728+0000", 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1109999999999999e-05 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045738+0000", 2024-04-24T14:34:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0016e-05 2024-04-24T14:34:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056183+0000", 2024-04-24T14:34:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010444836000000001 2024-04-24T14:34:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056470+0000", 2024-04-24T14:34:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028696899999999998 2024-04-24T14:34:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057160+0000", 2024-04-24T14:34:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069004699999999995 2024-04-24T14:34:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057291+0000", 2024-04-24T14:34:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013096200000000001 2024-04-24T14:34:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.061633+0000", 2024-04-24T14:34:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004341414 2024-04-24T14:34:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.864 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:45.864 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:45.864 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:45.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:45.866 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:45.868+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:45.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:34:45.966 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:45.970+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:45.971 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:45.971 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:45.971 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:45.971 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:45.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:46.067 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:46.072+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:46.079 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:46.079 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:46.079 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:46.080 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:46.080 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:46.082 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:46.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:46.102 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:46.122 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:46.144 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:46.168 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:46.170+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:46.168 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:46.182 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:46.182 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:46.190 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:46.192 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:46.192 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:46.192 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:46.192 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:46.193 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:46.193 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:46.193 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:46.209 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4] out_osds: [5, 1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 1] 2024-04-24T14:34:46.209 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:34:46.209 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-24T14:34:46.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T14:34:46.268 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:46.270+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:46.299 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:46.299 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:46.308 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:46.370 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:46.374+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:46.407 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:46.408 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:46.416 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:46.470 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:46.474+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:46.494 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:34:46.494 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:46.494 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 100, 2024-04-24T14:34:46.495 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "6706935e-d09a-472b-b839-321319fa9e54", 2024-04-24T14:34:46.495 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2024-04-24T14:32:09.059925+0000", 2024-04-24T14:34:46.495 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2024-04-24T14:34:44.828657+0000", 2024-04-24T14:34:46.495 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2024-04-24T14:34:00.856741+0000", 2024-04-24T14:34:46.495 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2024-04-24T14:34:41.175882+0000", 2024-04-24T14:34:46.495 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T14:34:46.495 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5801984, 2024-04-24T14:34:46.495 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2024-04-24T14:34:46.495 INFO:teuthology.orchestra.run.smithi090.stdout: "noscrub", 2024-04-24T14:34:46.495 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2024-04-24T14:34:46.495 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2024-04-24T14:34:46.495 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2024-04-24T14:34:46.495 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2024-04-24T14:34:46.495 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:34:46.496 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 5, 2024-04-24T14:34:46.496 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T14:34:46.496 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T14:34:46.496 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T14:34:46.496 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2024-04-24T14:34:46.496 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 4, 2024-04-24T14:34:46.496 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 6, 2024-04-24T14:34:46.496 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2024-04-24T14:34:46.496 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "jewel", 2024-04-24T14:34:46.496 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "quincy", 2024-04-24T14:34:46.496 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2024-04-24T14:34:46.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.496 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2024-04-24T14:34:46.496 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2024-04-24T14:34:46.497 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:21.233022+0000", 2024-04-24T14:34:46.497 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-24T14:34:46.497 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-24T14:34:46.497 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:34:46.497 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:34:46.497 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:34:46.497 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:34:46.497 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:34:46.497 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:34:46.497 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:34:46.497 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:34:46.497 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:34:46.497 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:34:46.498 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2024-04-24T14:34:46.498 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2024-04-24T14:34:46.498 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2024-04-24T14:34:46.498 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2024-04-24T14:34:46.498 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2024-04-24T14:34:46.498 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:34:46.498 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:34:46.498 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:34:46.498 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:34:46.498 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:34:46.498 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:34:46.498 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:34:46.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.498 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "14", 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:34:46.499 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:34:46.500 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:34:46.500 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:34:46.500 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:34:46.500 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:34:46.500 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:34:46.500 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:34:46.500 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:34:46.500 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:34:46.500 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:34:46.500 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:34:46.500 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:34:46.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.500 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:34:46.500 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:34:46.501 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:34:46.501 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:34:46.501 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:34:46.501 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:34:46.501 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:34:46.501 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:34:46.501 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:34:46.501 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:34:46.501 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:34:46.501 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-24T14:34:46.501 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2024-04-24T14:34:46.501 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2024-04-24T14:34:46.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.501 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:34:46.502 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2024-04-24T14:34:46.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.502 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-24T14:34:46.502 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2024-04-24T14:34:46.502 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:26.609964+0000", 2024-04-24T14:34:46.502 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-24T14:34:46.502 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T14:34:46.502 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:34:46.502 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:34:46.502 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:34:46.502 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:34:46.502 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:34:46.503 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:34:46.503 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:34:46.503 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:34:46.503 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:34:46.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:34:46.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2024-04-24T14:34:46.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2024-04-24T14:34:46.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2024-04-24T14:34:46.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2024-04-24T14:34:46.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2024-04-24T14:34:46.503 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:34:46.503 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:34:46.503 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "89", 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "21", 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "21", 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "21", 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 8, 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 89, 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:34:46.504 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:34:46.505 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:34:46.505 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [ 2024-04-24T14:34:46.505 INFO:teuthology.orchestra.run.smithi090.stdout: 3 2024-04-24T14:34:46.505 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:34:46.505 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:34:46.505 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": 3, 2024-04-24T14:34:46.505 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": 3, 2024-04-24T14:34:46.505 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:34:46.505 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:34:46.505 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:34:46.505 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:34:46.505 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:34:46.506 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:34:46.506 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:34:46.506 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:34:46.506 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:34:46.506 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:34:46.506 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:34:46.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.506 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:34:46.506 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:34:46.506 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:34:46.506 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:34:46.506 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:34:46.506 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:34:46.506 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:34:46.507 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:34:46.507 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:34:46.507 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:34:46.507 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:34:46.507 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:34:46.507 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:34:46.507 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:34:46.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.507 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-24T14:34:46.507 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "cache", 2024-04-24T14:34:46.507 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:33.987564+0000", 2024-04-24T14:34:46.507 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8201, 2024-04-24T14:34:46.508 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-04-24T14:34:46.508 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:34:46.508 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:34:46.508 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:34:46.508 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:34:46.508 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:34:46.508 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:34:46.508 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:34:46.508 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:34:46.508 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:34:46.508 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:34:46.508 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 4, 2024-04-24T14:34:46.508 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 4, 2024-04-24T14:34:46.508 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 4, 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 4, 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 4, 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "89", 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "21", 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "21", 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "21", 2024-04-24T14:34:46.509 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:34:46.510 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:34:46.510 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 8, 2024-04-24T14:34:46.510 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 89, 2024-04-24T14:34:46.510 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:34:46.510 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:34:46.510 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:34:46.510 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:34:46.510 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:34:46.510 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": 2, 2024-04-24T14:34:46.510 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:34:46.510 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:34:46.510 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "writeback", 2024-04-24T14:34:46.510 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:34:46.510 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 250, 2024-04-24T14:34:46.511 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:34:46.511 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:34:46.511 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:34:46.511 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:34:46.511 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:34:46.511 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:34:46.511 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:34:46.511 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bloom", 2024-04-24T14:34:46.511 INFO:teuthology.orchestra.run.smithi090.stdout: "false_positive_probability": 0.050000000000000003, 2024-04-24T14:34:46.511 INFO:teuthology.orchestra.run.smithi090.stdout: "target_size": 0, 2024-04-24T14:34:46.511 INFO:teuthology.orchestra.run.smithi090.stdout: "seed": 0 2024-04-24T14:34:46.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.511 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 60, 2024-04-24T14:34:46.511 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 8, 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [ 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000 2024-04-24T14:34:46.512 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:34:46.513 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:34:46.513 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:34:46.513 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:34:46.513 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:34:46.513 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:34:46.513 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:34:46.513 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.513 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 4, 2024-04-24T14:34:46.513 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "new_default_pool16809", 2024-04-24T14:34:46.513 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:33:35.603212+0000", 2024-04-24T14:34:46.513 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-24T14:34:46.513 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T14:34:46.514 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:34:46.514 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:34:46.514 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:34:46.514 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:34:46.514 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:34:46.514 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:34:46.514 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:34:46.514 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:34:46.514 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:34:46.514 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:34:46.514 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2024-04-24T14:34:46.514 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2024-04-24T14:34:46.514 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "53", 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:34:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 3, 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 53, 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:34:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:34:46.517 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:34:46.517 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:34:46.517 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:34:46.517 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:34:46.517 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:34:46.517 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:34:46.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.517 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:34:46.517 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:34:46.517 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:34:46.517 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:34:46.517 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:34:46.517 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:34:46.517 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:34:46.518 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:34:46.518 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:34:46.518 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:34:46.518 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:34:46.518 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:34:46.518 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:34:46.518 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:34:46.518 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.518 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.518 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:34:46.518 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-24T14:34:46.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.518 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "a2506b4f-3486-4f51-972f-1d9e4b015346", 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.484344482421875, 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 96, 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.519 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.520 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2024-04-24T14:34:46.520 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:34:46.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.520 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.520 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2024-04-24T14:34:46.520 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:34:46.520 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.520 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.520 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:34:46.520 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.520 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.521 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2024-04-24T14:34:46.521 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:34:46.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.521 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.521 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2024-04-24T14:34:46.521 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:34:46.521 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.521 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.521 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:34:46.521 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.522 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.522 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2024-04-24T14:34:46.522 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:34:46.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.522 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.522 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2024-04-24T14:34:46.522 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:34:46.522 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.522 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.522 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:34:46.522 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6817/20309", 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6819/20309", 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/20309", 2024-04-24T14:34:46.523 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6821/20309", 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "9e17324f-a615-494f-ba0d-7c8aa8dac95c", 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.2843475341796875, 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 11, 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 31, 2024-04-24T14:34:46.524 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 66, 2024-04-24T14:34:46.525 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 93, 2024-04-24T14:34:46.525 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 32, 2024-04-24T14:34:46.525 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:34:46.525 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:34:46.525 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.525 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.525 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2024-04-24T14:34:46.525 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:34:46.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.525 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.525 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2024-04-24T14:34:46.525 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:34:46.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.526 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.526 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:34:46.526 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.526 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.526 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2024-04-24T14:34:46.526 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:34:46.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.526 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.526 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2024-04-24T14:34:46.526 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:34:46.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.528 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.528 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:34:46.528 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.528 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.528 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2024-04-24T14:34:46.528 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:34:46.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.528 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.528 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2024-04-24T14:34:46.528 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:34:46.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.529 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.529 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/34413", 2024-04-24T14:34:46.529 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/34413", 2024-04-24T14:34:46.529 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/34413", 2024-04-24T14:34:46.529 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/34413", 2024-04-24T14:34:46.529 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:34:46.529 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:34:46.529 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:34:46.529 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.529 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-24T14:34:46.530 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7", 2024-04-24T14:34:46.530 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:34:46.530 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:34:46.530 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:34:46.530 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.8896331787109375, 2024-04-24T14:34:46.530 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:34:46.530 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:34:46.530 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:34:46.530 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 99, 2024-04-24T14:34:46.530 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:34:46.530 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:34:46.530 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:34:46.531 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.532 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.532 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2024-04-24T14:34:46.532 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:34:46.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.532 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.532 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2024-04-24T14:34:46.532 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:34:46.532 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.532 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.532 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:34:46.533 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.534 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2024-04-24T14:34:46.534 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:34:46.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.534 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2024-04-24T14:34:46.534 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:34:46.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.534 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.535 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6809/20310", 2024-04-24T14:34:46.535 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6811/20310", 2024-04-24T14:34:46.535 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6815/20310", 2024-04-24T14:34:46.535 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6813/20310", 2024-04-24T14:34:46.535 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:34:46.535 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:34:46.535 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:34:46.535 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.535 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-24T14:34:46.535 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "509c16da-22b2-48e8-aa51-c320c8c4fd8b", 2024-04-24T14:34:46.535 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 96, 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6800", 2024-04-24T14:34:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:34:46.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.537 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.537 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6801", 2024-04-24T14:34:46.537 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:34:46.537 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.537 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.537 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:34:46.537 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.537 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.537 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6802", 2024-04-24T14:34:46.537 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:34:46.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.538 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.538 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6803", 2024-04-24T14:34:46.538 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:34:46.538 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.538 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.538 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:34:46.538 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.538 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.538 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6806", 2024-04-24T14:34:46.538 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6807", 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6804", 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:34:46.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6805", 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6801/20316", 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6803/20316", 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6807/20316", 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6805/20316", 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1462066b-4e59-46d4-9b85-1f61ee1d56a6", 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 96, 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:34:46.541 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.542 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.542 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6808", 2024-04-24T14:34:46.542 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:34:46.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.542 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.542 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6809", 2024-04-24T14:34:46.542 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:34:46.542 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.542 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6810", 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6811", 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.543 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:34:46.544 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.544 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.544 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6814", 2024-04-24T14:34:46.544 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:34:46.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.544 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.544 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6815", 2024-04-24T14:34:46.544 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:34:46.544 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.544 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6812", 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6813", 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.545 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6809/20318", 2024-04-24T14:34:46.546 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6811/20318", 2024-04-24T14:34:46.546 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6815/20318", 2024-04-24T14:34:46.546 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6813/20318", 2024-04-24T14:34:46.546 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:34:46.546 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:34:46.546 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:34:46.546 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.546 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-24T14:34:46.546 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "4dc954a8-5f3c-4826-b9bc-98cf30519382", 2024-04-24T14:34:46.546 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:34:46.546 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2024-04-24T14:34:46.546 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2024-04-24T14:34:46.547 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:34:46.547 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:34:46.547 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:34:46.547 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:34:46.547 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 82, 2024-04-24T14:34:46.547 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:34:46.547 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:34:46.547 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:34:46.547 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.547 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.547 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6816", 2024-04-24T14:34:46.547 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:34:46.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6817", 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6818", 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6819", 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6822", 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.549 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.550 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6823", 2024-04-24T14:34:46.550 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:34:46.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.550 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.550 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:34:46.550 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:34:46.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.550 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:34:46.550 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6820", 2024-04-24T14:34:46.550 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:34:46.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.550 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:34:46.551 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6821", 2024-04-24T14:34:46.551 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:34:46.551 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.551 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.551 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6817/20317", 2024-04-24T14:34:46.551 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6819/20317", 2024-04-24T14:34:46.551 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6823/20317", 2024-04-24T14:34:46.551 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6821/20317", 2024-04-24T14:34:46.551 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:34:46.551 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:34:46.551 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:34:46.551 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:46.551 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.552 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:34:46.552 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2024-04-24T14:34:46.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.552 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-24T14:34:46.552 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:34:46.552 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:34:46.552 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:34:46.552 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:34:46.552 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:34:46.552 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.792899+0000", 2024-04-24T14:34:46.552 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:34:46.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.580414+0000", 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 32 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:34:46.553 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:34:46.554 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:34:46.554 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.713336+0000", 2024-04-24T14:34:46.554 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:34:46.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.554 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-24T14:34:46.554 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:34:46.554 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:34:46.554 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:34:46.554 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:34:46.554 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:34:46.554 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.579638+0000", 2024-04-24T14:34:46.554 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:34:46.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.555 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-24T14:34:46.555 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:34:46.555 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:34:46.555 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:34:46.555 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:34:46.555 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:34:46.555 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.570510+0000", 2024-04-24T14:34:46.555 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:34:46.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:46.555 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-24T14:34:46.555 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:34:46.555 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:34:46.556 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:34:46.556 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:34:46.556 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2024-04-24T14:34:46.556 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.736558+0000", 2024-04-24T14:34:46.556 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:34:46.556 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.556 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:34:46.556 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2024-04-24T14:34:46.556 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [], 2024-04-24T14:34:46.556 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2024-04-24T14:34:46.556 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2024-04-24T14:34:46.556 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2024-04-24T14:34:46.556 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2024-04-24T14:34:46.556 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2024-04-24T14:34:46.557 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2024-04-24T14:34:46.557 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-24T14:34:46.557 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-24T14:34:46.557 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-24T14:34:46.557 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-24T14:34:46.557 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2024-04-24T14:34:46.557 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:46.557 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [], 2024-04-24T14:34:46.557 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2024-04-24T14:34:46.557 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2024-04-24T14:34:46.557 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2024-04-24T14:34:46.557 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2024-04-24T14:34:46.557 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2024-04-24T14:34:46.558 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2024-04-24T14:34:46.558 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2024-04-24T14:34:46.558 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2024-04-24T14:34:46.558 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2024-04-24T14:34:46.558 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2024-04-24T14:34:46.558 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:46.558 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:46.560 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:46.561 INFO:tasks.thrashosds.thrasher:j is {'epoch': 100, 'fsid': '6706935e-d09a-472b-b839-321319fa9e54', 'created': '2024-04-24T14:32:09.059925+0000', 'modified': '2024-04-24T14:34:44.828657+0000', 'last_up_change': '2024-04-24T14:34:00.856741+0000', 'last_in_change': '2024-04-24T14:34:41.175882+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T14:32:21.233022+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T14:32:26.609964+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '21', 'last_force_op_resend_prenautilus': '21', 'last_force_op_resend_preluminous': '21', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 8, 'snap_epoch': 89, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-04-24T14:32:33.987564+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '21', 'last_force_op_resend_prenautilus': '21', 'last_force_op_resend_preluminous': '21', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 8, 'snap_epoch': 89, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 2, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'new_default_pool16809', 'create_time': '2024-04-24T14:33:35.603212+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 53, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'a2506b4f-3486-4f51-972f-1d9e4b015346', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.484344482421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 20309}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6819', 'nonce': 20309}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 20309}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6820', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 20309}]}, 'public_addr': '172.21.15.90:6817/20309', 'cluster_addr': '172.21.15.90:6819/20309', 'heartbeat_back_addr': '172.21.15.90:6823/20309', 'heartbeat_front_addr': '172.21.15.90:6821/20309', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9e17324f-a615-494f-ba0d-7c8aa8dac95c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.2843475341796875, 'last_clean_begin': 11, 'last_clean_end': 31, 'up_from': 66, 'up_thru': 93, 'down_at': 32, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 34413}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 34413}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 34413}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 34413}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 34413}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 34413}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 34413}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 34413}]}, 'public_addr': '172.21.15.90:6801/34413', 'cluster_addr': '172.21.15.90:6803/34413', 'heartbeat_back_addr': '172.21.15.90:6807/34413', 'heartbeat_front_addr': '172.21.15.90:6805/34413', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8896331787109375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 20310}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6810', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 20310}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6814', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 20310}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 20310}]}, 'public_addr': '172.21.15.90:6809/20310', 'cluster_addr': '172.21.15.90:6811/20310', 'heartbeat_back_addr': '172.21.15.90:6815/20310', 'heartbeat_front_addr': '172.21.15.90:6813/20310', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '509c16da-22b2-48e8-aa51-c320c8c4fd8b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6800', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6801', 'nonce': 20316}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6802', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6803', 'nonce': 20316}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6806', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6807', 'nonce': 20316}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6804', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6805', 'nonce': 20316}]}, 'public_addr': '172.21.15.103:6801/20316', 'cluster_addr': '172.21.15.103:6803/20316', 'heartbeat_back_addr': '172.21.15.103:6807/20316', 'heartbeat_front_addr': '172.21.15.103:6805/20316', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1462066b-4e59-46d4-9b85-1f61ee1d56a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6808', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6809', 'nonce': 20318}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6810', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6811', 'nonce': 20318}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6814', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6815', 'nonce': 20318}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6812', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6813', 'nonce': 20318}]}, 'public_addr': '172.21.15.103:6809/20318', 'cluster_addr': '172.21.15.103:6811/20318', 'heartbeat_back_addr': '172.21.15.103:6815/20318', 'heartbeat_front_addr': '172.21.15.103:6813/20318', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4dc954a8-5f3c-4826-b9bc-98cf30519382', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 82, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6816', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6817', 'nonce': 20317}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6818', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6819', 'nonce': 20317}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6822', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6823', 'nonce': 20317}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6820', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6821', 'nonce': 20317}]}, 'public_addr': '172.21.15.103:6817/20317', 'cluster_addr': '172.21.15.103:6819/20317', 'heartbeat_back_addr': '172.21.15.103:6823/20317', 'heartbeat_front_addr': '172.21.15.103:6821/20317', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.792899+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.580414+0000', 'dead_epoch': 32}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.713336+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.579638+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.570510+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.736558+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T14:34:46.561 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-24T14:34:46.561 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:46.571 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:46.572+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:46.661 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:46.661 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:46.670 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:46.671 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:46.674+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:46.771 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:46.772 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:46.772 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:46.776+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:46.781 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:46.872 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:46.874+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:46.876 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:46.878 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T14:34:46.885 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:46.973 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:46.976+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:46.980 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:46.980 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:46.989 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:47.075 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:47.080+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:47.090 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:47.090 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:47.099 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:47.175 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:47.180+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:47.237 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-24T14:34:47.246 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:47.246 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:47.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:47.276 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:47.278+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:47.357 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:47.357 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:47.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:47.377 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:47.380+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:47.466 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:47.466 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:47.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:47.477 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:47.478+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:47.577 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:47.577 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:47.578 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:47.580+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:47.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:47.679 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:47.682+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:47.685 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:47.686 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:47.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:47.781 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:47.782+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:47.795 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:47.795 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:47.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:47.881 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:47.884+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:47.904 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:47.904 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:47.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:47.982 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:47.986+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:48.012 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:48.012 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:48.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:48.083 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:48.088+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:48.121 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:48.122 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:48.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:48.184 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:48.186+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:48.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:34:48.228 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:48.228 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:48.237 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:48.286 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:48.290+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:48.335 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:48.345 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:48.345 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:48.355 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:48.356 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:48.375 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:48.386 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:48.390+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:48.396 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:48.418 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:48.439 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:48.442 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:48.442 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:48.442 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:48.443 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:48.443 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:48.443 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:48.453 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:48.453 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:48.453 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:48.453 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:48.453 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:48.462 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:48.487 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:48.492+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:48.563 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 145.94064077799999, 2024-04-24T14:34:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:34:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:34:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:34:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:34:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:34:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:34:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:34:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:34:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:34:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:34:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:34:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:34:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:34:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:34:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:34:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:34:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:34:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:34:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:34:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:34:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:34:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:34:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 145.94046369099999, 2024-04-24T14:34:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:34:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:34:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:34:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:34:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:34:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:34:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:34:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:34:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:34:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:34:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:34:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:34:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:34:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:34:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:34:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:34:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:34:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:34:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:34:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:34:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:34:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:34:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 145.940359777, 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:34:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:34:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:34:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:34:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:34:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:34:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:34:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:34:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:34:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:34:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:34:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:34:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:34:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:34:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:34:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:34:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:34:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:34:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:34:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:34:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 145.93997132600001, 2024-04-24T14:34:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:34:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:34:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:34:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:34:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:34:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:34:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:34:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:34:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:34:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:34:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:34:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:34:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:34:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:34:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:34:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:34:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:34:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:34:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:34:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:34:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:34:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:34:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 145.93988611399999, 2024-04-24T14:34:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:34:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:34:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:34:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:34:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:34:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:34:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:34:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:34:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:34:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:34:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:34:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:34:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:34:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:34:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:34:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:34:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:34:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:34:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:34:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:34:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:34:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:34:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 145.93964923300001, 2024-04-24T14:34:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:34:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:34:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:34:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:34:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:34:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:34:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:34:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:34:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:34:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:34:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:34:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:34:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:34:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:34:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:34:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 145.939567711, 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:34:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:34:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:34:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:34:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:34:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:34:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:34:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:34:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:34:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:34:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:34:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:34:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:34:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:34:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:34:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:34:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:34:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:34:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:34:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:34:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:34:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:34:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 145.50765536399999, 2024-04-24T14:34:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:34:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:34:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:34:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:34:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:34:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:34:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:34:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:34:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:34:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:34:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:34:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:34:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:34:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:34:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:34:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:34:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:34:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:34:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:34:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:34:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:34:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:34:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:34:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 145.50762706399999, 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:34:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:34:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:34:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:34:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:34:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:34:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:34:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:34:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:34:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:34:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:34:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:34:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:34:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:34:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:34:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:34:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:34:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:34:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:34:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:34:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:34:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 145.50753936000001, 2024-04-24T14:34:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:34:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:34:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:34:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:34:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:34:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:34:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:34:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:34:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:34:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:34:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:34:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:34:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:34:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:34:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:34:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:34:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:34:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:34:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:34:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 145.50741655900001, 2024-04-24T14:34:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:34:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:34:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:34:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:34:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:34:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:34:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:34:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:34:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:34:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:34:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:34:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:34:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:34:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:34:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:34:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:34:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:34:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:34:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:34:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:34:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:34:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:34:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:34:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:34:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 76.201171998999996, 2024-04-24T14:34:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:34:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:34:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:34:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:34:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:34:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:34:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:34:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:34:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:34:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:34:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:34:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:34:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:34:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:34:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:34:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:34:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:34:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:34:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:34:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:34:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:34:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:34:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.507047999999998, 2024-04-24T14:34:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:34:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:34:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:34:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:34:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:34:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:34:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:34:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:34:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:34:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:34:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:34:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:34:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:34:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:34:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:34:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:34:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:34:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:34:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:34:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 49.547688033999997, 2024-04-24T14:34:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:34:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:34:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:34:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:34:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:34:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:34:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:34:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:34:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:34:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:34:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:34:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:34:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:34:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:34:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 49.547607479, 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:34:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:34:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:34:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:34:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:34:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:34:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:34:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:34:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:34:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:34:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:34:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:34:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:34:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:34:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:34:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:34:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 49.547559749999998, 2024-04-24T14:34:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:34:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:34:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:34:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:34:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:34:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:34:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:34:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 49.547531986999999, 2024-04-24T14:34:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:34:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:34:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:34:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:34:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:34:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:34:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:34:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:34:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:34:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:34:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:34:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:34:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:34:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:34:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:34:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:34:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:34:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:34:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:34:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:34:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:34:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:34:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 49.547492552000001, 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:34:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:34:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:34:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:34:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:34:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:34:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:34:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:34:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:34:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:34:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:34:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:34:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 49.547467568000002, 2024-04-24T14:34:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:34:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:34:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:34:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:34:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:34:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:34:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:34:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:34:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:34:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:34:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:34:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:34:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:34:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:34:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:34:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:34:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:34:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 49.547441909, 2024-04-24T14:34:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:34:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:34:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:34:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:34:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:34:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:34:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:34:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:34:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:34:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:34:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:34:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:34:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:34:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:34:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:34:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:34:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:34:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:34:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:34:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:48.683 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:48.685 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:48.685 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:48.690+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:48.786 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:48.790+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:48.832 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:48.832 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:48.832 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:48.832 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:48.841 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:48.887 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:48.888+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:48.942 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:48.942 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:48.951 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:48.987 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:48.990+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:49.056 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 67.395695558, 2024-04-24T14:34:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:34:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:34:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:34:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:34:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:34:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:34:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:34:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:34:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:34:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:34:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:34:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:34:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:34:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:34:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 67.392661032000007, 2024-04-24T14:34:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:34:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:34:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:34:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:34:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:34:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:34:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:34:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:34:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:34:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:34:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:34:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:34:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:34:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:34:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 67.392636846000002, 2024-04-24T14:34:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:34:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:34:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:34:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:34:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:34:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:34:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:34:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:34:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:34:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:34:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:34:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:34:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:34:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:34:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 67.392619461999999, 2024-04-24T14:34:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:34:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:34:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:34:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:34:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:34:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:34:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:34:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:34:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:34:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:34:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:34:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:34:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:34:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:34:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:34:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 67.378183370000002, 2024-04-24T14:34:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:34:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:34:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:34:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:34:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:34:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:34:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:34:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:34:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:34:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:34:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:34:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:34:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:34:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:34:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:34:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 67.378151071000005, 2024-04-24T14:34:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:34:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:34:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:34:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:34:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:34:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:34:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:34:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:34:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:34:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:34:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:34:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:34:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:34:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:34:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 67.363898793999994, 2024-04-24T14:34:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:34:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:34:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:34:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:34:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:34:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:34:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:34:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:34:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:34:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:34:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:34:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:34:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:34:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:34:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:34:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 67.363869694000002, 2024-04-24T14:34:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:34:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:34:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:34:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:34:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:34:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:34:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:34:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:34:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:34:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:34:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:34:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:34:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:34:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:34:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 67.363843763999995, 2024-04-24T14:34:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:34:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:34:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:34:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:34:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:34:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:34:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:34:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:34:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:34:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:34:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:34:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:34:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:34:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:34:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 67.348123299999997, 2024-04-24T14:34:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:34:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:34:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:34:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:34:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:34:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:34:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:34:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:34:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:34:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:34:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:34:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:34:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:34:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:34:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:34:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 67.348091367999999, 2024-04-24T14:34:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:34:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:34:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:34:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:34:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:34:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:34:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:34:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 67.348023851999997, 2024-04-24T14:34:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:34:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:34:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:34:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:34:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:34:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:34:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:34:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:34:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:34:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:34:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:34:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 67.290610968999999, 2024-04-24T14:34:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:34:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:34:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:34:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:34:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:34:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:34:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:34:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:34:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:34:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:34:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:34:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:34:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:34:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:34:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:34:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:103 3.1 e87/74 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head v 87'261, mlcod=87'258)", 2024-04-24T14:34:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.099329765, 2024-04-24T14:34:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019745728000000001, 2024-04-24T14:34:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:49.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960679+0000", 2024-04-24T14:34:49.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:49.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960692+0000", 2024-04-24T14:34:49.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3709999999999999e-05 2024-04-24T14:34:49.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960722+0000", 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9791999999999999e-05 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960733+0000", 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.128e-05 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960857+0000", 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000123981 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960926+0000", 2024-04-24T14:34:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8881999999999995e-05 2024-04-24T14:34:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980353+0000", 2024-04-24T14:34:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019426853000000001 2024-04-24T14:34:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980426+0000", 2024-04-24T14:34:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2803000000000001e-05 2024-04-24T14:34:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:105 3.1 e87/74 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head v 87'263, mlcod=87'258)", 2024-04-24T14:34:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.090203231, 2024-04-24T14:34:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014864743999999999, 2024-04-24T14:34:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969805+0000", 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969820+0000", 2024-04-24T14:34:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5116999999999999e-05 2024-04-24T14:34:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969852+0000", 2024-04-24T14:34:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2135999999999999e-05 2024-04-24T14:34:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969864+0000", 2024-04-24T14:34:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.189e-05 2024-04-24T14:34:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972122+0000", 2024-04-24T14:34:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022580389999999999 2024-04-24T14:34:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972165+0000", 2024-04-24T14:34:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2429000000000003e-05 2024-04-24T14:34:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984619+0000", 2024-04-24T14:34:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012454735 2024-04-24T14:34:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984672+0000", 2024-04-24T14:34:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2077999999999998e-05 2024-04-24T14:34:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:112 3.1 e87/74 3:a983798d:::rbd_data.12ff64619e2d.0000000000000006:head v 87'265, mlcod=87'258)", 2024-04-24T14:34:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.079864395000001, 2024-04-24T14:34:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012821978, 2024-04-24T14:34:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980144+0000", 2024-04-24T14:34:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980159+0000", 2024-04-24T14:34:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4769999999999999e-05 2024-04-24T14:34:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980189+0000", 2024-04-24T14:34:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0419000000000001e-05 2024-04-24T14:34:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980213+0000", 2024-04-24T14:34:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4036000000000002e-05 2024-04-24T14:34:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980452+0000", 2024-04-24T14:34:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00023921500000000001 2024-04-24T14:34:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980468+0000", 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5843999999999999e-05 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992932+0000", 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012463314 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992968+0000", 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6075999999999998e-05 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:113 3.1 e87/74 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head v 87'267, mlcod=87'258)", 2024-04-24T14:34:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.076329671, 2024-04-24T14:34:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010561022999999999, 2024-04-24T14:34:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983679+0000", 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983699+0000", 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0392000000000001e-05 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983737+0000", 2024-04-24T14:34:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8340000000000002e-05 2024-04-24T14:34:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983756+0000", 2024-04-24T14:34:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8402000000000001e-05 2024-04-24T14:34:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984706+0000", 2024-04-24T14:34:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095024400000000005 2024-04-24T14:34:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984741+0000", 2024-04-24T14:34:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4721000000000002e-05 2024-04-24T14:34:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994211+0000", 2024-04-24T14:34:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094708559999999997 2024-04-24T14:34:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994241+0000", 2024-04-24T14:34:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9941e-05 2024-04-24T14:34:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:115 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.075875633999999, 2024-04-24T14:34:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010347242, 2024-04-24T14:34:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984132+0000", 2024-04-24T14:34:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984142+0000", 2024-04-24T14:34:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7370000000000001e-06 2024-04-24T14:34:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984154+0000", 2024-04-24T14:34:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2288e-05 2024-04-24T14:34:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984163+0000", 2024-04-24T14:34:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3749999999999992e-06 2024-04-24T14:34:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984246+0000", 2024-04-24T14:34:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2962999999999998e-05 2024-04-24T14:34:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984473+0000", 2024-04-24T14:34:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00022696399999999999 2024-04-24T14:34:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984771+0000", 2024-04-24T14:34:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00029739499999999999 2024-04-24T14:34:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992998+0000", 2024-04-24T14:34:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082275750000000009 2024-04-24T14:34:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994299+0000", 2024-04-24T14:34:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013005289999999999 2024-04-24T14:34:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994403+0000", 2024-04-24T14:34:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010376499999999999 2024-04-24T14:34:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994482+0000", 2024-04-24T14:34:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9013999999999999e-05 2024-04-24T14:34:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:128 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.054684706, 2024-04-24T14:34:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011784144, 2024-04-24T14:34:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:34:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 128 2024-04-24T14:34:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005324+0000", 2024-04-24T14:34:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005332+0000", 2024-04-24T14:34:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2989999999999997e-06 2024-04-24T14:34:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005354+0000", 2024-04-24T14:34:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1705999999999999e-05 2024-04-24T14:34:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005362+0000", 2024-04-24T14:34:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5599999999999996e-06 2024-04-24T14:34:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005418+0000", 2024-04-24T14:34:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6685999999999999e-05 2024-04-24T14:34:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005535+0000", 2024-04-24T14:34:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117008 2024-04-24T14:34:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005732+0000", 2024-04-24T14:34:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019718700000000001 2024-04-24T14:34:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016440+0000", 2024-04-24T14:34:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010707787999999999 2024-04-24T14:34:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016913+0000", 2024-04-24T14:34:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047270400000000001 2024-04-24T14:34:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017047+0000", 2024-04-24T14:34:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013434699999999999 2024-04-24T14:34:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017109+0000", 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2192000000000004e-05 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:130 3.1 e87/74 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 87'269, mlcod=87'265)", 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 22.054231946000002, 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013945631999999999, 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005777+0000", 2024-04-24T14:34:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005791+0000", 2024-04-24T14:34:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3743999999999999e-05 2024-04-24T14:34:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005803+0000", 2024-04-24T14:34:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2330999999999999e-05 2024-04-24T14:34:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005810+0000", 2024-04-24T14:34:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8990000000000004e-06 2024-04-24T14:34:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005888+0000", 2024-04-24T14:34:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7872000000000003e-05 2024-04-24T14:34:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005911+0000", 2024-04-24T14:34:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3156e-05 2024-04-24T14:34:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019642+0000", 2024-04-24T14:34:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013730456 2024-04-24T14:34:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019724+0000", 2024-04-24T14:34:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1981e-05 2024-04-24T14:34:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.148 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:49.150 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:49.151 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:49.154+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:49.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T14:34:49.251 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:49.254+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:49.253 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:49.253 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:49.253 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:49.253 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:49.263 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:49.352 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:49.356+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:49.358 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:49.358 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:49.358 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:49.358 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:49.358 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:49.368 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:49.453 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:49.454+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:49.516 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:34:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.449388280999997, 2024-04-24T14:34:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:34:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:34:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:34:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:34:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:34:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:34:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:34:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:34:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:34:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:34:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:34:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:34:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:34:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:34:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:34:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:34:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:34:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.454573701999998, 2024-04-24T14:34:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:34:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:34:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:34:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:34:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:34:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:34:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:34:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:34:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:34:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:34:49.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:34:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:34:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.443644444, 2024-04-24T14:34:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:34:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:34:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:34:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:34:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:34:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:34:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:34:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:34:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:34:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:34:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:34:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:34:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.432688362999997, 2024-04-24T14:34:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:34:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:34:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:34:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:34:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:34:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:34:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:34:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:34:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:34:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:34:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:34:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:34:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:34:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:34:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:34:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:34:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:34:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.421118997999997, 2024-04-24T14:34:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:34:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:34:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:34:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:34:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:34:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:34:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:34:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:34:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:34:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:34:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:34:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:34:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.167999338999998, 2024-04-24T14:34:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:34:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:34:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:34:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:34:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:34:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:34:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:34:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:34:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:34:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:34:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:34:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:34:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:34:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.157254170999998, 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:34:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:34:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:34:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:34:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:34:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:34:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:34:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:34:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:34:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:34:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:34:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:34:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:34:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:34:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:34:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:34:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:34:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.144887894999997, 2024-04-24T14:34:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:34:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:34:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:34:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:34:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:34:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:34:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:34:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:34:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:34:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:34:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:34:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:34:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:34:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:34:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:34:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:34:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:34:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.019951773999999, 2024-04-24T14:34:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:34:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:34:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:34:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:34:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:34:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:34:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:34:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:34:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.000563931000002, 2024-04-24T14:34:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:34:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:34:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:34:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:34:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:34:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:34:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:34:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:34:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:34:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:34:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:34:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:34:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:34:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:34:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:34:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:34:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:34:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.990836088000002, 2024-04-24T14:34:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:34:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:34:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:34:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:34:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:34:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:34:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:34:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:34:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:34:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:34:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:34:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:34:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:34:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:34:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:34:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:34:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:34:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.830726407999997, 2024-04-24T14:34:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:34:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:34:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:34:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:34:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:34:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:34:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:34:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:34:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:34:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:34:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:34:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.820997411999997, 2024-04-24T14:34:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:34:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:34:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:34:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:34:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:34:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:34:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:34:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:34:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:34:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:34:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:34:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 50.486026576999997, 2024-04-24T14:34:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:34:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:34:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:34:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:34:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:34:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:34:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:34:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:34:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:34:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:34:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:34:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:34:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:34:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:34:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:34:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.641875829, 2024-04-24T14:34:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:34:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:34:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:34:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:34:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:34:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:34:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:5 3.2 3.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:34:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.313638187000002, 2024-04-24T14:34:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043717790000000001, 2024-04-24T14:34:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-24T14:34:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:34:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:34:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 5 2024-04-24T14:34:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207678+0000", 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999962 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207700+0000", 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1313000000000001e-05 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207757+0000", 2024-04-24T14:34:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7499e-05 2024-04-24T14:34:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207779+0000", 2024-04-24T14:34:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1478999999999999e-05 2024-04-24T14:34:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.212054+0000", 2024-04-24T14:34:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042750849999999996 2024-04-24T14:34:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.418112829000002, 2024-04-24T14:34:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044084859999999997, 2024-04-24T14:34:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103206+0000", 2024-04-24T14:34:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104731+0000", 2024-04-24T14:34:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001524755 2024-04-24T14:34:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104759+0000", 2024-04-24T14:34:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7973000000000001e-05 2024-04-24T14:34:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104807+0000", 2024-04-24T14:34:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8112000000000003e-05 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104907+0000", 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100317 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104918+0000", 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747999999999999e-05 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.107616+0000", 2024-04-24T14:34:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026977559999999999 2024-04-24T14:34:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(3.2 83/82 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.411718451999999, 2024-04-24T14:34:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041425760000000002, 2024-04-24T14:34:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109600+0000", 2024-04-24T14:34:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111210+0000", 2024-04-24T14:34:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016097279999999999 2024-04-24T14:34:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111237+0000", 2024-04-24T14:34:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7189999999999999e-05 2024-04-24T14:34:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111251+0000", 2024-04-24T14:34:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.365e-05 2024-04-24T14:34:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111367+0000", 2024-04-24T14:34:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000115703 2024-04-24T14:34:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111378+0000", 2024-04-24T14:34:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1873e-05 2024-04-24T14:34:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.113744+0000", 2024-04-24T14:34:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023657069999999999 2024-04-24T14:34:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.394817876000001, 2024-04-24T14:34:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046193859999999996, 2024-04-24T14:34:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126501+0000", 2024-04-24T14:34:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128169+0000", 2024-04-24T14:34:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016686229999999999 2024-04-24T14:34:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128202+0000", 2024-04-24T14:34:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2539000000000001e-05 2024-04-24T14:34:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128222+0000", 2024-04-24T14:34:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0149e-05 2024-04-24T14:34:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128375+0000", 2024-04-24T14:34:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000153459 2024-04-24T14:34:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128399+0000", 2024-04-24T14:34:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3393000000000001e-05 2024-04-24T14:34:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.131122+0000", 2024-04-24T14:34:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027227140000000002 2024-04-24T14:34:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 49'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@49'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 31.373036496000001, 2024-04-24T14:34:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038414529999999999, 2024-04-24T14:34:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148282+0000", 2024-04-24T14:34:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149875+0000", 2024-04-24T14:34:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015929900000000001 2024-04-24T14:34:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149908+0000", 2024-04-24T14:34:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3102000000000003e-05 2024-04-24T14:34:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149941+0000", 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2480000000000001e-05 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150031+0000", 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0394000000000002e-05 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150043+0000", 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1967000000000001e-05 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.152125+0000", 2024-04-24T14:34:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020817029999999999 2024-04-24T14:34:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:49.601 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:49.602 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:49.603 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:49.606+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:49.703 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:49.703 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:49.703 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:49.708+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:49.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:49.805 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:49.806+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:49.808 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:49.808 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:49.808 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:49.809 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:49.809 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:49.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:49.906 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:49.908+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:49.964 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.088893220999999, 2024-04-24T14:34:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:34:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:49.964 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:34:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:49.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:34:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:34:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:34:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:34:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:34:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:34:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:34:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:34:49.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:34:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:34:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:34:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:34:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:34:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:49.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:34:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:34:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:34:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:34:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:34:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:34:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:34:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:34:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.088864235999999, 2024-04-24T14:34:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:34:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:49.970 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:34:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:49.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:34:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:34:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:34:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:34:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:34:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:34:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:34:49.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:34:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:34:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:34:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:34:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:34:49.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:34:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:34:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.088845751000001, 2024-04-24T14:34:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:34:49.976 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:34:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:34:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:34:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:34:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:34:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:34:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:34:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:34:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:34:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:34:49.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:34:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:34:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:34:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:34:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:34:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:34:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:34:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:34:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:34:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:34:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.088830469999998, 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:34:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:34:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:34:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:34:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:34:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:34:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:34:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:34:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:34:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:34:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:34:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:34:49.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:34:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:34:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:34:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:34:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:49.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.088814689000003, 2024-04-24T14:34:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:34:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:34:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.988 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:49.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:34:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:34:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:34:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:34:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:34:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:34:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:34:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:34:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:34:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:34:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:34:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:34:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:34:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:34:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.088800401, 2024-04-24T14:34:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:34:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:34:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:49.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:49.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:34:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:34:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:34:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:34:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:49.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:34:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:34:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:34:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:34:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:34:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:34:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:34:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:34:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:34:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:34:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:34:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:34:49.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:34:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:34:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:34:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:34:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:49.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.088780649, 2024-04-24T14:34:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:34:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:34:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.000 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:50.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:34:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:34:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:34:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:34:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:34:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:34:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:34:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:34:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:34:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:34:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:34:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:34:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:34:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:34:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:34:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:34:50.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:34:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:34:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:34:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:34:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.088763202000003, 2024-04-24T14:34:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:34:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:34:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.006 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:34:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:34:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:34:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:34:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:34:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:34:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:34:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:34:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:34:50.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:34:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:34:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:34:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:34:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:34:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:34:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:34:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:34:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:34:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:34:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:34:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.011 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:100 3.2 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.214708387000002, 2024-04-24T14:34:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009078015, 2024-04-24T14:34:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.012 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 100 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754019+0000", 2024-04-24T14:34:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754025+0000", 2024-04-24T14:34:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2110000000000003e-06 2024-04-24T14:34:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754037+0000", 2024-04-24T14:34:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1868999999999999e-05 2024-04-24T14:34:50.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754046+0000", 2024-04-24T14:34:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7430000000000008e-06 2024-04-24T14:34:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754095+0000", 2024-04-24T14:34:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8924000000000001e-05 2024-04-24T14:34:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754153+0000", 2024-04-24T14:34:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8165000000000003e-05 2024-04-24T14:34:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754262+0000", 2024-04-24T14:34:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108694 2024-04-24T14:34:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.762999+0000", 2024-04-24T14:34:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087372590000000007 2024-04-24T14:34:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763027+0000", 2024-04-24T14:34:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8467e-05 2024-04-24T14:34:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763070+0000", 2024-04-24T14:34:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2236e-05 2024-04-24T14:34:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763098+0000", 2024-04-24T14:34:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8561999999999999e-05 2024-04-24T14:34:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:101 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [delete] snapc 7=[7,6,4] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.009500720999998, 2024-04-24T14:34:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012894743, 2024-04-24T14:34:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 101 2024-04-24T14:34:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959227+0000", 2024-04-24T14:34:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959233+0000", 2024-04-24T14:34:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.857e-06 2024-04-24T14:34:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959249+0000", 2024-04-24T14:34:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6155999999999999e-05 2024-04-24T14:34:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959257+0000", 2024-04-24T14:34:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7440000000000007e-06 2024-04-24T14:34:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959790+0000", 2024-04-24T14:34:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053381100000000003 2024-04-24T14:34:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960206+0000", 2024-04-24T14:34:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041549700000000002 2024-04-24T14:34:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960744+0000", 2024-04-24T14:34:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053762199999999997 2024-04-24T14:34:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971963+0000", 2024-04-24T14:34:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011219816000000001 2024-04-24T14:34:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972021+0000", 2024-04-24T14:34:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7855000000000001e-05 2024-04-24T14:34:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972073+0000", 2024-04-24T14:34:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2196e-05 2024-04-24T14:34:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972123+0000", 2024-04-24T14:34:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9311999999999998e-05 2024-04-24T14:34:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:102 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.009460695000001, 2024-04-24T14:34:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012939713, 2024-04-24T14:34:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 102 2024-04-24T14:34:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959267+0000", 2024-04-24T14:34:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959274+0000", 2024-04-24T14:34:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.593e-06 2024-04-24T14:34:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959291+0000", 2024-04-24T14:34:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6387000000000001e-05 2024-04-24T14:34:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959301+0000", 2024-04-24T14:34:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0023e-05 2024-04-24T14:34:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969594+0000", 2024-04-24T14:34:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010293182 2024-04-24T14:34:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969902+0000", 2024-04-24T14:34:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030835299999999998 2024-04-24T14:34:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970408+0000", 2024-04-24T14:34:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050588100000000002 2024-04-24T14:34:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972009+0000", 2024-04-24T14:34:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016011210000000001 2024-04-24T14:34:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972132+0000", 2024-04-24T14:34:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122521 2024-04-24T14:34:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972174+0000", 2024-04-24T14:34:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2242e-05 2024-04-24T14:34:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972208+0000", 2024-04-24T14:34:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3522999999999997e-05 2024-04-24T14:34:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:104 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.008849480999999, 2024-04-24T14:34:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014589062, 2024-04-24T14:34:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 104 2024-04-24T14:34:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959877+0000", 2024-04-24T14:34:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:50.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959890+0000", 2024-04-24T14:34:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2537e-05 2024-04-24T14:34:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959909+0000", 2024-04-24T14:34:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9340000000000001e-05 2024-04-24T14:34:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959922+0000", 2024-04-24T14:34:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2882e-05 2024-04-24T14:34:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970900+0000", 2024-04-24T14:34:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010977736 2024-04-24T14:34:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971039+0000", 2024-04-24T14:34:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139593 2024-04-24T14:34:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971453+0000", 2024-04-24T14:34:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041351099999999998 2024-04-24T14:34:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974044+0000", 2024-04-24T14:34:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025908289999999998 2024-04-24T14:34:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974377+0000", 2024-04-24T14:34:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033310399999999997 2024-04-24T14:34:50.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974433+0000", 2024-04-24T14:34:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6166999999999999e-05 2024-04-24T14:34:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974468+0000", 2024-04-24T14:34:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5336000000000002e-05 2024-04-24T14:34:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:106 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.008790147999999, 2024-04-24T14:34:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017192064999999999, 2024-04-24T14:34:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 106 2024-04-24T14:34:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959938+0000", 2024-04-24T14:34:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959943+0000", 2024-04-24T14:34:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.677e-06 2024-04-24T14:34:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959953+0000", 2024-04-24T14:34:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8069999999999994e-06 2024-04-24T14:34:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:34:50.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0560000000000001e-06 2024-04-24T14:34:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972232+0000", 2024-04-24T14:34:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012272460000000001 2024-04-24T14:34:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972304+0000", 2024-04-24T14:34:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2630000000000001e-05 2024-04-24T14:34:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972548+0000", 2024-04-24T14:34:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000244047 2024-04-24T14:34:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976900+0000", 2024-04-24T14:34:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043514069999999998 2024-04-24T14:34:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977076+0000", 2024-04-24T14:34:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176281 2024-04-24T14:34:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977106+0000", 2024-04-24T14:34:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0015e-05 2024-04-24T14:34:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977131+0000", 2024-04-24T14:34:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4669e-05 2024-04-24T14:34:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:114 3.2 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.985468984000001, 2024-04-24T14:34:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090880089999999993, 2024-04-24T14:34:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 114 2024-04-24T14:34:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983259+0000", 2024-04-24T14:34:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983264+0000", 2024-04-24T14:34:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9130000000000002e-06 2024-04-24T14:34:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983277+0000", 2024-04-24T14:34:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3723999999999999e-05 2024-04-24T14:34:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983282+0000", 2024-04-24T14:34:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0429999999999998e-06 2024-04-24T14:34:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983339+0000", 2024-04-24T14:34:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6323000000000003e-05 2024-04-24T14:34:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983464+0000", 2024-04-24T14:34:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012554299999999999 2024-04-24T14:34:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983693+0000", 2024-04-24T14:34:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000228848 2024-04-24T14:34:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.986978+0000", 2024-04-24T14:34:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032852810000000001 2024-04-24T14:34:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992288+0000", 2024-04-24T14:34:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053095369999999996 2024-04-24T14:34:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992322+0000", 2024-04-24T14:34:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3683999999999998e-05 2024-04-24T14:34:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992348+0000", 2024-04-24T14:34:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6157999999999999e-05 2024-04-24T14:34:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:121 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.973412884999998, 2024-04-24T14:34:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098798159999999996, 2024-04-24T14:34:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 121 2024-04-24T14:34:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995315+0000", 2024-04-24T14:34:50.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995320+0000", 2024-04-24T14:34:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0239999999999997e-06 2024-04-24T14:34:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995328+0000", 2024-04-24T14:34:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7800000000000001e-06 2024-04-24T14:34:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995334+0000", 2024-04-24T14:34:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2330000000000003e-06 2024-04-24T14:34:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998810+0000", 2024-04-24T14:34:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034756090000000002 2024-04-24T14:34:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998870+0000", 2024-04-24T14:34:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0625999999999999e-05 2024-04-24T14:34:50.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.999091+0000", 2024-04-24T14:34:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000220631 2024-04-24T14:34:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.001389+0000", 2024-04-24T14:34:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022977549999999998 2024-04-24T14:34:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005097+0000", 2024-04-24T14:34:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037085439999999998 2024-04-24T14:34:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005154+0000", 2024-04-24T14:34:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6208000000000001e-05 2024-04-24T14:34:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005196+0000", 2024-04-24T14:34:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2052000000000003e-05 2024-04-24T14:34:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.055 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:127 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.963111369, 2024-04-24T14:34:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013414918, 2024-04-24T14:34:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.056 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2024-04-24T14:34:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005616+0000", 2024-04-24T14:34:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005622+0000", 2024-04-24T14:34:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2020000000000003e-06 2024-04-24T14:34:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005632+0000", 2024-04-24T14:34:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2024-04-24T14:34:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005637+0000", 2024-04-24T14:34:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7439999999999998e-06 2024-04-24T14:34:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005947+0000", 2024-04-24T14:34:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000310792 2024-04-24T14:34:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006059+0000", 2024-04-24T14:34:50.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011192399999999999 2024-04-24T14:34:50.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:50.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006503+0000", 2024-04-24T14:34:50.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000443394 2024-04-24T14:34:50.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018864+0000", 2024-04-24T14:34:50.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012361162 2024-04-24T14:34:50.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018877+0000", 2024-04-24T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3497e-05 2024-04-24T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018965+0000", 2024-04-24T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7476000000000001e-05 2024-04-24T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019032+0000", 2024-04-24T14:34:50.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7469000000000001e-05 2024-04-24T14:34:50.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.062 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.062 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:129 3.2 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.062 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:50.062 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.962971136, 2024-04-24T14:34:50.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010059754000000001, 2024-04-24T14:34:50.062 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.062 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.063 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 129 2024-04-24T14:34:50.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.063 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:50.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:50.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005757+0000", 2024-04-24T14:34:50.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:50.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005760+0000", 2024-04-24T14:34:50.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2430000000000001e-06 2024-04-24T14:34:50.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005769+0000", 2024-04-24T14:34:50.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4810000000000006e-06 2024-04-24T14:34:50.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005773+0000", 2024-04-24T14:34:50.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:50.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005840+0000", 2024-04-24T14:34:50.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6934999999999995e-05 2024-04-24T14:34:50.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005961+0000", 2024-04-24T14:34:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012153 2024-04-24T14:34:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006181+0000", 2024-04-24T14:34:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021947500000000001 2024-04-24T14:34:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015520+0000", 2024-04-24T14:34:50.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093393049999999991 2024-04-24T14:34:50.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015733+0000", 2024-04-24T14:34:50.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021355699999999999 2024-04-24T14:34:50.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015773+0000", 2024-04-24T14:34:50.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9554000000000001e-05 2024-04-24T14:34:50.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015817+0000", 2024-04-24T14:34:50.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4325e-05 2024-04-24T14:34:50.068 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.068 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.068 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.068 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:131 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.068 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:50.068 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.962693255000001, 2024-04-24T14:34:50.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014251976, 2024-04-24T14:34:50.068 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.068 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.069 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.069 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.069 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.069 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131 2024-04-24T14:34:50.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.069 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:50.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:50.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006034+0000", 2024-04-24T14:34:50.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:50.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006042+0000", 2024-04-24T14:34:50.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2970000000000004e-06 2024-04-24T14:34:50.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006061+0000", 2024-04-24T14:34:50.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8505999999999999e-05 2024-04-24T14:34:50.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006071+0000", 2024-04-24T14:34:50.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0409e-05 2024-04-24T14:34:50.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007899+0000", 2024-04-24T14:34:50.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018274789999999999 2024-04-24T14:34:50.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008281+0000", 2024-04-24T14:34:50.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038244700000000001 2024-04-24T14:34:50.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:50.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008528+0000", 2024-04-24T14:34:50.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024723900000000001 2024-04-24T14:34:50.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019070+0000", 2024-04-24T14:34:50.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010541867999999999 2024-04-24T14:34:50.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020099+0000", 2024-04-24T14:34:50.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00102852 2024-04-24T14:34:50.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020226+0000", 2024-04-24T14:34:50.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012744300000000001 2024-04-24T14:34:50.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020287+0000", 2024-04-24T14:34:50.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1253000000000005e-05 2024-04-24T14:34:50.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.074 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.074 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:132 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.074 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:50.074 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.962639402000001, 2024-04-24T14:34:50.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015917981000000001, 2024-04-24T14:34:50.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.075 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 132 2024-04-24T14:34:50.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.075 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:50.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:50.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:50.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006093+0000", 2024-04-24T14:34:50.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3819999999999997e-06 2024-04-24T14:34:50.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006103+0000", 2024-04-24T14:34:50.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0352e-05 2024-04-24T14:34:50.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006110+0000", 2024-04-24T14:34:50.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3910000000000003e-06 2024-04-24T14:34:50.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015586+0000", 2024-04-24T14:34:50.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094759119999999995 2024-04-24T14:34:50.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015730+0000", 2024-04-24T14:34:50.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014487199999999999 2024-04-24T14:34:50.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:50.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015886+0000", 2024-04-24T14:34:50.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015573799999999999 2024-04-24T14:34:50.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020306+0000", 2024-04-24T14:34:50.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044199399999999998 2024-04-24T14:34:50.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021887+0000", 2024-04-24T14:34:50.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015809400000000001 2024-04-24T14:34:50.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021942+0000", 2024-04-24T14:34:50.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5071000000000002e-05 2024-04-24T14:34:50.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.022007+0000", 2024-04-24T14:34:50.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5110000000000005e-05 2024-04-24T14:34:50.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.080 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.080 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.5324.0:127 3.0 e87/69 ondisk, result = 0)", 2024-04-24T14:34:50.081 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:50.081 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.960930334, 2024-04-24T14:34:50.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011245125, 2024-04-24T14:34:50.081 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.081 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.081 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:50.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:50.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:50.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007804+0000", 2024-04-24T14:34:50.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6610000000000003e-06 2024-04-24T14:34:50.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007817+0000", 2024-04-24T14:34:50.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3030999999999999e-05 2024-04-24T14:34:50.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007832+0000", 2024-04-24T14:34:50.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.47e-05 2024-04-24T14:34:50.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018843+0000", 2024-04-24T14:34:50.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011010647 2024-04-24T14:34:50.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018853+0000", 2024-04-24T14:34:50.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319000000000001e-05 2024-04-24T14:34:50.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019043+0000", 2024-04-24T14:34:50.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019055600000000001 2024-04-24T14:34:50.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.084 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.085 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.085 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:50.086 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:50.090+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:50.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:50.187 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:50.192+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:50.194 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:50.194 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:50.195 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:50.195 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:50.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:50.248 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-24T14:34:50.288 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:50.290+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:50.313 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:50.313 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:50.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:50.389 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:50.390+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:50.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:34:50.469 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:50.469 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.715429252000007, 2024-04-24T14:34:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:34:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.470 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:34:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:34:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:34:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:34:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:34:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:34:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:34:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:34:50.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:34:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:34:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:34:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:34:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:34:50.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:34:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:34:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:34:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:34:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:34:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:34:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:34:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:34:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:34:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:50.476 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.715378418, 2024-04-24T14:34:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:34:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:34:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:34:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:34:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:34:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:34:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:34:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:34:50.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:34:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:34:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:34:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:34:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:34:50.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:34:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:34:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:34:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:34:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:34:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:34:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:34:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:34:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:34:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:50.482 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.715342110999998, 2024-04-24T14:34:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:34:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:34:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:50.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:34:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:34:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:34:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:34:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:34:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:34:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:34:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:34:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:34:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:34:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:34:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:34:50.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:34:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:34:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:34:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:34:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:34:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:34:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:34:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:34:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:34:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:50.488 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.711930860999999, 2024-04-24T14:34:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:34:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:34:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:34:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:34:50.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:34:50.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:34:50.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:50.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:34:50.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:34:50.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:34:50.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:34:50.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:50.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:34:50.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:34:50.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:34:50.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:34:50.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:34:50.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:34:50.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:34:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:34:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.711869183999994, 2024-04-24T14:34:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:34:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.494 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:50.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:50.495 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:34:50.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.495 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:50.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:50.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:34:50.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:34:50.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:34:50.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:34:50.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:34:50.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:34:50.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:34:50.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:34:50.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:34:50.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:50.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:34:50.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:34:50.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:34:50.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:34:50.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:50.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:34:50.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:34:50.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:50.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:34:50.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:34:50.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:34:50.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:34:50.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:34:50.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.500 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.500 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:50.500 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:50.500 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.711834608000004, 2024-04-24T14:34:50.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:34:50.500 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.500 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:50.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:50.501 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:34:50.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.501 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:50.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:50.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:50.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:50.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:34:50.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:34:50.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:34:50.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:34:50.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:34:50.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:34:50.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:34:50.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:34:50.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:50.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:34:50.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:34:50.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.506 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:50.506 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:50.506 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.711805268999996, 2024-04-24T14:34:50.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:34:50.506 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.506 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.506 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.506 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:50.506 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:50.506 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:34:50.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.506 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:34:50.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:34:50.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:50.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:34:50.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:34:50.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:34:50.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:34:50.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:50.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:34:50.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:34:50.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:34:50.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:34:50.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:34:50.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:34:50.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:34:50.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:34:50.511 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.511 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.511 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.511 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:50.511 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:50.511 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.711728687000004, 2024-04-24T14:34:50.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:50.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:34:50.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:34:50.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:34:50.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:34:50.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:34:50.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:34:50.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:34:50.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:34:50.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:34:50.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.516 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.517 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:50.517 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:50.517 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.711681142000003, 2024-04-24T14:34:50.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:34:50.517 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.517 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.517 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.517 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:50.517 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:50.517 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:34:50.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.517 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:34:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:34:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:34:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:34:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:34:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:34:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:34:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:34:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:34:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.711640500000001, 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:34:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:34:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:34:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:34:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:34:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:34:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:34:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:34:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:34:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:34:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:34:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.711623388999996, 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:34:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:34:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:34:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:34:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:34:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:34:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:34:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:34:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:34:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:34:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:34:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:103 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:50.533 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.512057276, 2024-04-24T14:34:50.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032646351999999997, 2024-04-24T14:34:50.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.534 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.534 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.534 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.534 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.534 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 103 2024-04-24T14:34:50.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.534 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:50.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959674+0000", 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959688+0000", 2024-04-24T14:34:50.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959722+0000", 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3936000000000002e-05 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959735+0000", 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2926e-05 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959921+0000", 2024-04-24T14:34:50.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185601 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960324+0000", 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040356700000000002 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960839+0000", 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051410599999999996 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.975634+0000", 2024-04-24T14:34:50.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014795770999999999 2024-04-24T14:34:50.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992113+0000", 2024-04-24T14:34:50.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016478982999999999 2024-04-24T14:34:50.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992228+0000", 2024-04-24T14:34:50.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114295 2024-04-24T14:34:50.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992323+0000", 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5344000000000001e-05 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:105 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.511807498, 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032581757000000003, 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.539 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959925+0000", 2024-04-24T14:34:50.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:50.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959937+0000", 2024-04-24T14:34:50.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2391999999999999e-05 2024-04-24T14:34:50.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:34:50.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2535999999999999e-05 2024-04-24T14:34:50.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959970+0000", 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1039999999999999e-05 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969531+0000", 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095603949999999993 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969669+0000", 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013806499999999999 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:50.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970010+0000", 2024-04-24T14:34:50.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034062200000000001 2024-04-24T14:34:50.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976983+0000", 2024-04-24T14:34:50.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069735600000000002 2024-04-24T14:34:50.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992413+0000", 2024-04-24T14:34:50.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015429841999999999 2024-04-24T14:34:50.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992457+0000", 2024-04-24T14:34:50.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4169999999999999e-05 2024-04-24T14:34:50.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992508+0000", 2024-04-24T14:34:50.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0995000000000002e-05 2024-04-24T14:34:50.544 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.544 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.544 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.545 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:113 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.545 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:50.545 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.491581206999999, 2024-04-24T14:34:50.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014793863000000001, 2024-04-24T14:34:50.545 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.545 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.545 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.545 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.545 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.545 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 113 2024-04-24T14:34:50.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.545 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980152+0000", 2024-04-24T14:34:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980156+0000", 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.619e-06 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980163+0000", 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7070000000000003e-06 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980173+0000", 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0957e-05 2024-04-24T14:34:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983381+0000", 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032079930000000001 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983628+0000", 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024665700000000001 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983964+0000", 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000336332 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992694+0000", 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087297009999999994 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994878+0000", 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021841439999999998 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994917+0000", 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8544999999999998e-05 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994947+0000", 2024-04-24T14:34:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9705000000000001e-05 2024-04-24T14:34:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:130 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.465887771999999, 2024-04-24T14:34:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014646289999999999, 2024-04-24T14:34:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 130 2024-04-24T14:34:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005844+0000", 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005853+0000", 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1319999999999994e-06 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005870+0000", 2024-04-24T14:34:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7448999999999998e-05 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005876+0000", 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3319999999999997e-06 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005913+0000", 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6835999999999998e-05 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006017+0000", 2024-04-24T14:34:50.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010358500000000001 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006139+0000", 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001227 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.010194+0000", 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040549380000000001 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020365+0000", 2024-04-24T14:34:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010170983 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020428+0000", 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2457000000000002e-05 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020492+0000", 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4592999999999996e-05 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:155 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1601536~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.432226985, 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015144391, 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 155 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039506+0000", 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039766+0000", 2024-04-24T14:34:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025994400000000002 2024-04-24T14:34:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039780+0000", 2024-04-24T14:34:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3366e-05 2024-04-24T14:34:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039787+0000", 2024-04-24T14:34:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4359999999999996e-06 2024-04-24T14:34:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.050886+0000", 2024-04-24T14:34:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011098993 2024-04-24T14:34:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051026+0000", 2024-04-24T14:34:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001395 2024-04-24T14:34:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051211+0000", 2024-04-24T14:34:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018547099999999999 2024-04-24T14:34:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054490+0000", 2024-04-24T14:34:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032785499999999999 2024-04-24T14:34:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054505+0000", 2024-04-24T14:34:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5906999999999999e-05 2024-04-24T14:34:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054594+0000", 2024-04-24T14:34:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8817e-05 2024-04-24T14:34:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054651+0000", 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7040999999999998e-05 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:156 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1732608~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.431936278999999, 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016895194999999998, 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 156 2024-04-24T14:34:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039797+0000", 2024-04-24T14:34:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040065+0000", 2024-04-24T14:34:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000268361 2024-04-24T14:34:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040086+0000", 2024-04-24T14:34:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1024000000000001e-05 2024-04-24T14:34:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040094+0000", 2024-04-24T14:34:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9379999999999992e-06 2024-04-24T14:34:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051936+0000", 2024-04-24T14:34:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011842518 2024-04-24T14:34:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052062+0000", 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125482 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052206+0000", 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143953 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054715+0000", 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025088240000000002 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056534+0000", 2024-04-24T14:34:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001819436 2024-04-24T14:34:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056610+0000", 2024-04-24T14:34:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5383000000000007e-05 2024-04-24T14:34:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056693+0000", 2024-04-24T14:34:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3283999999999997e-05 2024-04-24T14:34:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:157 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1863680~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.4316289, 2024-04-24T14:34:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016763674999999999, 2024-04-24T14:34:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157 2024-04-24T14:34:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040104+0000", 2024-04-24T14:34:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040244+0000", 2024-04-24T14:34:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014035499999999999 2024-04-24T14:34:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040255+0000", 2024-04-24T14:34:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0135999999999999e-05 2024-04-24T14:34:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040260+0000", 2024-04-24T14:34:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7919999999999998e-06 2024-04-24T14:34:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053406+0000", 2024-04-24T14:34:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013146023999999999 2024-04-24T14:34:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053531+0000", 2024-04-24T14:34:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012471800000000001 2024-04-24T14:34:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053715+0000", 2024-04-24T14:34:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018395099999999999 2024-04-24T14:34:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056711+0000", 2024-04-24T14:34:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029956520000000001 2024-04-24T14:34:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056783+0000", 2024-04-24T14:34:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2010999999999999e-05 2024-04-24T14:34:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056831+0000", 2024-04-24T14:34:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8189999999999998e-05 2024-04-24T14:34:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056869+0000", 2024-04-24T14:34:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7812000000000003e-05 2024-04-24T14:34:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:158 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1994752~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.431466972999999, 2024-04-24T14:34:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016845334999999999, 2024-04-24T14:34:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2024-04-24T14:34:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040266+0000", 2024-04-24T14:34:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040379+0000", 2024-04-24T14:34:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112728 2024-04-24T14:34:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040387+0000", 2024-04-24T14:34:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8499999999999994e-06 2024-04-24T14:34:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040391+0000", 2024-04-24T14:34:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2719999999999999e-06 2024-04-24T14:34:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054803+0000", 2024-04-24T14:34:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014411548 2024-04-24T14:34:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054982+0000", 2024-04-24T14:34:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017941699999999999 2024-04-24T14:34:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.055167+0000", 2024-04-24T14:34:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018542100000000001 2024-04-24T14:34:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056718+0000", 2024-04-24T14:34:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015505670000000001 2024-04-24T14:34:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057020+0000", 2024-04-24T14:34:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030227500000000002 2024-04-24T14:34:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057070+0000", 2024-04-24T14:34:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9611000000000003e-05 2024-04-24T14:34:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057112+0000", 2024-04-24T14:34:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2373999999999998e-05 2024-04-24T14:34:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:159 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.426031985000002, 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015930650000000001, 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045701+0000", 2024-04-24T14:34:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045707+0000", 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.562e-06 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045728+0000", 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1109999999999999e-05 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045738+0000", 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0016e-05 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056183+0000", 2024-04-24T14:34:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010444836000000001 2024-04-24T14:34:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056470+0000", 2024-04-24T14:34:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028696899999999998 2024-04-24T14:34:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057160+0000", 2024-04-24T14:34:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069004699999999995 2024-04-24T14:34:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057291+0000", 2024-04-24T14:34:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013096200000000001 2024-04-24T14:34:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.061633+0000", 2024-04-24T14:34:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004341414 2024-04-24T14:34:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.584 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.584 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:50.584 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:50.584 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:50.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:50.586 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:50.586 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:50.588+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:50.604 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:50.627 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:50.649 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:50.671 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:50.685 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:50.688+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:50.693 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:50.695 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:50.695 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:50.696 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:50.696 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:50.696 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:50.696 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:50.705 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:50.705 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:50.715 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:50.786 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:50.788+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:50.856 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:50.856 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:50.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:50.887 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:50.890+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:50.964 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:50.964 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:50.973 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:51.014 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:51.018+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:51.084 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:51.085 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:51.089 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:51.092+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:51.094 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:51.190 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:51.192+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:51.195 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:51.195 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:51.204 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:51.291 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:51.294+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:51.350 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:51.350 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:51.360 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:51.393 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:51.394+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:51.456 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:51.456 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:51.465 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:51.494 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:51.498+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:51.562 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4] out_osds: [5, 1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 1] 2024-04-24T14:34:51.562 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:34:51.562 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-24T14:34:51.562 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-24T14:34:51.562 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-24T14:34:51.563 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-04-24T14:34:51.563 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:51.563 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:51.571 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:51.594 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:51.596+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:51.662 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:51.662 INFO:teuthology.orchestra.run.smithi103.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-24T14:34:51.662 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:51.665 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:51.666 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:51.675 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:51.695 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:51.698+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:51.796 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:51.800+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:51.818 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:51.818 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:51.827 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:51.896 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:51.898+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:51.925 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:51.934 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:51.997 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:52.000+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:52.037 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:52.037 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:52.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:52.098 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:52.102+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:52.154 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:52.154 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:52.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:52.199 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:52.202+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:52.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T14:34:52.263 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:52.264 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:52.275 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:52.322 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:52.326+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:52.401 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:52.404+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:52.422 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:52.423 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:52.432 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:52.503 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:52.506+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:52.526 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:52.526 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:52.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:52.603 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:52.606+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:52.677 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:52.677 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:52.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:52.704 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:52.708+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:52.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:34:52.788 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:52.788 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:52.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:52.805 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:52.808+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:52.855 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:52.875 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:52.896 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:52.897 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:52.906 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:52.908+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:52.906 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:52.919 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:52.943 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:52.972 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:52.974 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:52.974 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:52.974 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:52.974 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:52.975 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:52.975 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:53.002 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:53.002 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:53.002 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:53.002 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:53.002 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:53.007 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:53.008+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:53.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:53.108 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:53.110+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:53.160 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:53.160 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:53.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:53.160 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:53.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.161 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:53.161 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:53.161 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.692560211, 2024-04-24T14:34:53.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:34:53.161 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.161 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.161 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.161 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.161 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.162 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:34:53.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.162 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:53.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:53.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:34:53.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:53.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:34:53.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:34:53.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:34:53.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:34:53.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:34:53.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:34:53.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:34:53.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:34:53.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:34:53.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:34:53.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:34:53.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:34:53.166 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.166 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.167 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.167 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.167 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:53.167 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.317203508000006, 2024-04-24T14:34:53.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:34:53.167 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.167 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.167 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.167 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.167 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.168 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:34:53.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.168 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:53.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:53.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:34:53.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:53.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:34:53.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:34:53.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:34:53.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:34:53.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:34:53.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:53.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:34:53.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:34:53.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:34:53.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:34:53.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:34:53.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:34:53.173 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.173 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.173 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.173 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.173 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:53.173 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.315775353999996, 2024-04-24T14:34:53.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:34:53.173 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.174 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.174 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:34:53.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.174 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:53.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:53.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:34:53.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:53.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:34:53.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:34:53.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:34:53.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:34:53.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:34:53.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:34:53.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:34:53.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:34:53.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:34:53.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:34:53.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:34:53.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:34:53.179 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.179 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.179 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.179 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.179 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:53.179 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.315748264999996, 2024-04-24T14:34:53.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:34:53.180 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.180 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.180 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:34:53.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.180 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:53.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:53.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:34:53.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:53.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:34:53.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:34:53.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:34:53.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:34:53.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:34:53.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:34:53.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:34:53.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:34:53.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:34:53.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:34:53.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:34:53.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:34:53.184 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.185 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.185 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.185 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.185 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:53.185 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.315724298000006, 2024-04-24T14:34:53.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:34:53.185 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.186 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.186 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.186 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.186 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.186 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:34:53.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.186 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:53.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:53.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:34:53.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:53.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:34:53.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:34:53.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:34:53.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:34:53.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:34:53.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:34:53.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:34:53.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:34:53.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:34:53.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:34:53.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:34:53.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:34:53.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.191 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.191 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.191 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:53.191 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.301740319000004, 2024-04-24T14:34:53.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:34:53.191 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.191 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.192 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:34:53.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.192 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:53.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:53.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:34:53.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:53.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:34:53.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:34:53.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:34:53.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:34:53.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:34:53.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:34:53.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:34:53.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:34:53.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:34:53.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:34:53.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:34:53.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:34:53.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:34:53.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:34:53.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:34:53.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:34:53.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:34:53.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:34:53.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:34:53.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:34:53.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.198 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.199 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.199 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:53.199 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.301624008000005, 2024-04-24T14:34:53.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:34:53.199 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.199 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.199 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.199 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.200 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.200 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:34:53.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.200 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:53.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:53.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:34:53.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:53.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:34:53.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:34:53.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:34:53.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:34:53.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:34:53.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:34:53.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:34:53.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:34:53.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:34:53.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:34:53.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:34:53.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:34:53.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:34:53.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:34:53.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:34:53.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:34:53.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:34:53.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:34:53.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:34:53.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:34:53.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.206 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.301576350000005, 2024-04-24T14:34:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:34:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:34:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:34:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:34:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:34:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:34:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:34:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:34:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:34:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:34:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:34:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:34:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:34:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:34:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:34:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:34:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:34:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:34:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:34:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:34:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:34:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:34:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:34:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.297539052000005, 2024-04-24T14:34:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:34:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.216 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:34:53.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.216 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:34:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:34:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:34:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:34:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:34:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:34:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:34:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:34:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:34:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:34:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:34:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:34:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:34:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:34:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:34:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:34:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:34:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:34:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:34:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:34:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.297475293000005, 2024-04-24T14:34:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:34:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:34:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:34:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:34:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:34:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:34:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:34:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:34:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:34:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:34:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:34:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:34:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:34:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:34:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:34:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:34:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:34:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:34:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:34:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:34:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:34:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:34:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.297449803000006, 2024-04-24T14:34:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:34:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:34:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:34:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:34:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:34:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:34:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:34:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:34:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:34:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:34:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:34:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:34:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:34:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:34:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:34:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:34:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:34:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:34:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:34:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:34:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:34:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.297426052999995, 2024-04-24T14:34:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:34:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:34:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:34:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:34:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:34:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:34:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:34:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:34:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:34:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:34:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:34:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:34:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:34:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:34:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:34:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:34:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:34:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:34:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:34:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:34:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:34:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:34:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:34:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.297387540000003, 2024-04-24T14:34:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:34:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:34:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:34:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:34:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:34:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:34:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:34:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:34:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:34:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:34:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:34:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:34:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:34:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:34:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:34:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:34:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:34:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:34:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:34:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:34:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:34:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.297359342999997, 2024-04-24T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:34:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:34:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:34:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:34:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:34:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:34:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:34:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:34:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:34:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:34:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:34:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:34:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:34:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:34:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:34:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:34:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:34:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:34:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:34:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:34:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:34:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.297337885999994, 2024-04-24T14:34:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:34:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:34:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:34:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:34:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:34:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:34:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:34:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:34:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:34:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:34:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:34:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:34:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:34:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:34:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:34:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:34:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:34:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:34:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:34:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:34:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:34:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:34:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.297319044000005, 2024-04-24T14:34:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:34:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:34:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:34:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:34:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:34:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:34:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:34:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:34:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:34:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:34:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:34:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:34:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:34:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:34:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:34:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:34:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:34:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:34:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:34:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:34:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:34:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:34:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:34:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.084486894999998, 2024-04-24T14:34:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:34:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:34:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:34:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:34:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:34:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:34:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:34:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:34:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:34:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:34:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:34:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:34:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:34:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:34:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:34:53.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:34:53.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:34:53.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:34:53.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:34:53.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:34:53.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:34:53.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:34:53.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:34:53.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.287 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.287 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:39 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1753088~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:53.287 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:53.287 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.144714479000001, 2024-04-24T14:34:53.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01139394, 2024-04-24T14:34:53.287 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.287 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:53.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:53.288 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 39 2024-04-24T14:34:53.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.288 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:53.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:53.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018967+0000", 2024-04-24T14:34:53.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:53.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019242+0000", 2024-04-24T14:34:53.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027504500000000002 2024-04-24T14:34:53.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019274+0000", 2024-04-24T14:34:53.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2472999999999999e-05 2024-04-24T14:34:53.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019288+0000", 2024-04-24T14:34:53.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3283e-05 2024-04-24T14:34:53.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026753+0000", 2024-04-24T14:34:53.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074651120000000003 2024-04-24T14:34:53.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026971+0000", 2024-04-24T14:34:53.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021810800000000001 2024-04-24T14:34:53.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:53.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.027211+0000", 2024-04-24T14:34:53.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023975 2024-04-24T14:34:53.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:53.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030200+0000", 2024-04-24T14:34:53.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029890709999999998 2024-04-24T14:34:53.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:53.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030252+0000", 2024-04-24T14:34:53.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1926999999999999e-05 2024-04-24T14:34:53.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:53.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030312+0000", 2024-04-24T14:34:53.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0602e-05 2024-04-24T14:34:53.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030363+0000", 2024-04-24T14:34:53.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0342e-05 2024-04-24T14:34:53.295 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.295 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.295 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.295 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:40 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1884160~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:53.295 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:53.295 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.144378504999999, 2024-04-24T14:34:53.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011199064, 2024-04-24T14:34:53.296 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.296 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.296 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.296 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:53.296 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:53.296 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 40 2024-04-24T14:34:53.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.296 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:53.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:53.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019303+0000", 2024-04-24T14:34:53.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:53.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019729+0000", 2024-04-24T14:34:53.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042620899999999998 2024-04-24T14:34:53.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019769+0000", 2024-04-24T14:34:53.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9847e-05 2024-04-24T14:34:53.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019797+0000", 2024-04-24T14:34:53.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8347000000000002e-05 2024-04-24T14:34:53.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028299+0000", 2024-04-24T14:34:53.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085010629999999997 2024-04-24T14:34:53.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028385+0000", 2024-04-24T14:34:53.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6824999999999996e-05 2024-04-24T14:34:53.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:53.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028510+0000", 2024-04-24T14:34:53.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012483000000000001 2024-04-24T14:34:53.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:53.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030410+0000", 2024-04-24T14:34:53.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001899612 2024-04-24T14:34:53.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:53.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030416+0000", 2024-04-24T14:34:53.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7699999999999998e-06 2024-04-24T14:34:53.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:53.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030461+0000", 2024-04-24T14:34:53.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5157999999999997e-05 2024-04-24T14:34:53.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030504+0000", 2024-04-24T14:34:53.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2988999999999998e-05 2024-04-24T14:34:53.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.303 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.303 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:41 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 2015232~81920] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:53.303 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:53.303 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.143869798000001, 2024-04-24T14:34:53.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012345257, 2024-04-24T14:34:53.303 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:53.303 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:53.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:53.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:53.304 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 41 2024-04-24T14:34:53.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.304 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:53.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:53.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:53.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:53.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:53.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:53.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:53.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019812+0000", 2024-04-24T14:34:53.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:53.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:53.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019975+0000", 2024-04-24T14:34:53.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016325700000000001 2024-04-24T14:34:53.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:53.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020025+0000", 2024-04-24T14:34:53.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9484999999999998e-05 2024-04-24T14:34:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020035+0000", 2024-04-24T14:34:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0444e-05 2024-04-24T14:34:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029150+0000", 2024-04-24T14:34:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091150160000000001 2024-04-24T14:34:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029352+0000", 2024-04-24T14:34:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020166499999999999 2024-04-24T14:34:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029611+0000", 2024-04-24T14:34:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025949600000000001 2024-04-24T14:34:53.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030518+0000", 2024-04-24T14:34:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090633300000000005 2024-04-24T14:34:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032088+0000", 2024-04-24T14:34:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015703290000000001 2024-04-24T14:34:53.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032130+0000", 2024-04-24T14:34:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2273e-05 2024-04-24T14:34:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032159+0000", 2024-04-24T14:34:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8387000000000001e-05 2024-04-24T14:34:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.310 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.311 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.311 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:53.311 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:53.311 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:53.312 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:53.316+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:53.313 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:53.314 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-24T14:34:53.413 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:53.414+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:53.421 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:53.421 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:53.421 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:53.422 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:53.431 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:53.515 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:53.518+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:53.530 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:53.531 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:53.531 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:53.531 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:53.531 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:53.540 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:53.615 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:53.620+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:53.637 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:53.637 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:53.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:53.637 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:53.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.637 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:53.637 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 151.01592339600001, 2024-04-24T14:34:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:34:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:34:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:34:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:34:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:34:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:34:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:34:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:34:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:34:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:34:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:34:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:34:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:34:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:34:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:34:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:34:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:34:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:34:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:34:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:34:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:34:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:34:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:34:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 151.01574630900001, 2024-04-24T14:34:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:34:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:34:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:34:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:34:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:34:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:34:53.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:34:53.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:34:53.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:34:53.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:34:53.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:34:53.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:34:53.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:34:53.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:53.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:34:53.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:34:53.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:53.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:34:53.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:34:53.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:53.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:34:53.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:34:53.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:53.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:34:53.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:34:53.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:34:53.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:34:53.650 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.650 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.650 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.650 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:53.650 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:53.650 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 151.01564239499999, 2024-04-24T14:34:53.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:34:53.651 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.651 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.651 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.651 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:53.651 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:53.651 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:34:53.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.651 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:53.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:53.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:34:53.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:53.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:34:53.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:34:53.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:34:53.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:34:53.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:34:53.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:53.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:34:53.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:34:53.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:34:53.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:34:53.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:53.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:34:53.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:34:53.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:53.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:34:53.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:34:53.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:53.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:34:53.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:34:53.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:53.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:34:53.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:34:53.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:34:53.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:34:53.656 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.656 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.656 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.657 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:53.657 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:53.657 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 151.01525394399999, 2024-04-24T14:34:53.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:34:53.657 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.657 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.657 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.657 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:53.657 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:53.657 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:34:53.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.658 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:53.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.658 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.658 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:53.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:34:53.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:53.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:34:53.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:34:53.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:34:53.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:34:53.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.660 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.660 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:34:53.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:34:53.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.660 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.660 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:34:53.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:34:53.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.660 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.660 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:34:53.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:34:53.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:53.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:34:53.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:34:53.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:53.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:34:53.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:34:53.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:53.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:34:53.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:34:53.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:53.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:34:53.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:34:53.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:34:53.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:34:53.663 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.663 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.663 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.663 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:53.663 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:53.663 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 151.01516873200001, 2024-04-24T14:34:53.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:34:53.663 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.663 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.664 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.664 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:53.664 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:53.664 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:53.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.664 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:53.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:53.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:34:53.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:53.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:34:53.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:34:53.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:34:53.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:34:53.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:34:53.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:34:53.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:34:53.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:34:53.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:34:53.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:34:53.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:53.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:34:53.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:34:53.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:53.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:34:53.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:34:53.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:53.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:34:53.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:34:53.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:53.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:34:53.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:34:53.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:34:53.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:34:53.669 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.669 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.669 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.669 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:53.669 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:53.669 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 151.014931851, 2024-04-24T14:34:53.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:34:53.670 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.670 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.670 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.670 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:53.670 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:53.670 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:34:53.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.670 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:53.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:53.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:34:53.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:53.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:34:53.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:34:53.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:34:53.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:34:53.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:34:53.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:34:53.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:34:53.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:34:53.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:34:53.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:34:53.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:53.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:34:53.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:34:53.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:53.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:34:53.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:34:53.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:53.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:34:53.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:34:53.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:53.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:34:53.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:34:53.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:34:53.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:34:53.675 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.675 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.676 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.676 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:53.676 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:53.676 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 151.01485032900001, 2024-04-24T14:34:53.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:34:53.676 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.676 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.676 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.676 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:53.676 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:53.677 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:34:53.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.677 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:53.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:53.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:34:53.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:53.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:34:53.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:34:53.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:34:53.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:34:53.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:34:53.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:34:53.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:34:53.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:34:53.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:34:53.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:34:53.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:53.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:34:53.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:34:53.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:53.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:34:53.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:34:53.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:53.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:34:53.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:34:53.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:53.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:34:53.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:34:53.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:34:53.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:34:53.682 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.682 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.682 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.682 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:53.682 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:53.682 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 150.582937982, 2024-04-24T14:34:53.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:34:53.682 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.683 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.683 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.683 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:53.683 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:53.683 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:34:53.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.683 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.683 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:53.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:53.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:34:53.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:53.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:34:53.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:34:53.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:34:53.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:34:53.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:34:53.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:34:53.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:34:53.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:34:53.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:34:53.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:34:53.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:53.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:34:53.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:34:53.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:53.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:34:53.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:34:53.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:53.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:34:53.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:34:53.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:53.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:34:53.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:34:53.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.688 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.688 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:34:53.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:34:53.688 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.689 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.689 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.689 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:53.689 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:53.689 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 150.58290968200001, 2024-04-24T14:34:53.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:34:53.689 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.689 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.689 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.689 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:53.689 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:53.690 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:34:53.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.690 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:53.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:53.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:34:53.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:53.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:34:53.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:34:53.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:34:53.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:34:53.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:34:53.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:34:53.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:34:53.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:34:53.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:34:53.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:34:53.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:53.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:34:53.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:34:53.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:53.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:34:53.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:34:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:34:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:34:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:34:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:34:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.694 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:34:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:34:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 150.582821978, 2024-04-24T14:34:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:34:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.695 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:34:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.696 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:34:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:34:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:34:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.697 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:34:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:34:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:34:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:34:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.698 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:34:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:34:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:34:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:34:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:34:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:34:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.700 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:53.700 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:34:53.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:34:53.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.700 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:53.700 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:34:53.700 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:34:53.700 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.700 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.700 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:53.700 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:34:53.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:34:53.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.701 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.701 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:34:53.701 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:34:53.701 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.701 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.701 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.701 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.701 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:53.701 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:53.702 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 150.58269917699999, 2024-04-24T14:34:53.702 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:34:53.702 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.702 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.702 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.702 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:53.702 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:53.702 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:34:53.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.702 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.702 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.702 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:53.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:53.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.703 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.703 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:34:53.703 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:53.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.703 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.704 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:34:53.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:34:53.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.704 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.704 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:34:53.704 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:34:53.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.704 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.704 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:34:53.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:34:53.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:34:53.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:34:53.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.705 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.705 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.705 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:34:53.705 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:34:53.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:53.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:34:53.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:34:53.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.706 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:53.706 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:34:53.706 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:34:53.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:53.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:34:53.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:34:53.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:53.707 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:34:53.707 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:34:53.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.707 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.708 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:34:53.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:34:53.708 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.708 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.708 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.708 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:34:53.708 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:53.708 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.276454616999999, 2024-04-24T14:34:53.708 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:34:53.708 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.708 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:53.709 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.709 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:34:53.709 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:34:53.709 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:34:53.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.709 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.709 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:53.709 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.709 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:53.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:34:53.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:53.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.710 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.710 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:34:53.710 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:34:53.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:34:53.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:34:53.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.711 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:34:53.711 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:34:53.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.711 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:34:53.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:34:53.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:34:53.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:34:53.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:34:53.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.712 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.712 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:34:53.712 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:34:53.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.713 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:53.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:34:53.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:34:53.713 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.713 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.713 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:34:53.713 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:34:53.713 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.713 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.714 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.714 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.714 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.714 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:53.714 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:53.714 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 73.582330618, 2024-04-24T14:34:53.714 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:34:53.714 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.714 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:53.714 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.714 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:34:53.714 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:34:53.715 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:34:53.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.715 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.715 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:53.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.715 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.715 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.715 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:53.715 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:34:53.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:53.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.716 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:34:53.716 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:34:53.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.716 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.716 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:34:53.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:34:53.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:34:53.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:34:53.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.717 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.717 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.717 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:34:53.717 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:34:53.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.718 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:34:53.718 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:34:53.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.718 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.718 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:34:53.718 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:34:53.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.718 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.718 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.719 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:34:53.719 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:34:53.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.719 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.719 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.719 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:34:53.719 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:34:53.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.719 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.719 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.719 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:34:53.719 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:34:53.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.720 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.720 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.720 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:34:53.720 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:34:53.720 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.720 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.720 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.720 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.720 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.720 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:53.720 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:53.720 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.622970651999999, 2024-04-24T14:34:53.720 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:34:53.721 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.721 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:53.721 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.721 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:53.721 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:53.721 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:34:53.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.721 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.721 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.721 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.721 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:53.722 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.722 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.722 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:53.722 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.722 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.722 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.722 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:34:53.722 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:53.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:34:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:34:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:34:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:34:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:34:53.723 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:34:53.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.724 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.724 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.724 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:34:53.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:34:53.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.724 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.724 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.724 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:34:53.724 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:34:53.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.724 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.725 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.725 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:34:53.725 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:34:53.725 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.725 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.725 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.725 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:34:53.725 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:34:53.725 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.725 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.725 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.725 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:34:53.725 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:34:53.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.726 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:34:53.726 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:34:53.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.726 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.726 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.726 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:34:53.726 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:34:53.726 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.726 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.726 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.727 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.727 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:53.727 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:53.727 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.622890097000003, 2024-04-24T14:34:53.727 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:34:53.727 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.727 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:53.727 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.727 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:53.727 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:53.727 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:34:53.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.728 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.728 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.728 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.728 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:53.728 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.728 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.728 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.728 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:53.728 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.728 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.728 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.729 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:34:53.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:53.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.729 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.729 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.729 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:34:53.729 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:34:53.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.729 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.729 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.729 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:34:53.730 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:34:53.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.730 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.730 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.730 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:34:53.730 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:34:53.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.730 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.730 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.730 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:34:53.730 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:34:53.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.731 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.731 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:34:53.731 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:34:53.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.731 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.731 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:34:53.731 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:34:53.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.731 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.731 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:34:53.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:34:53.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:34:53.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:34:53.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.732 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.732 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.732 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:34:53.732 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:34:53.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.733 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.733 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:34:53.733 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:34:53.733 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.733 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.733 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.733 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.733 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:53.733 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:53.733 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.622842368000001, 2024-04-24T14:34:53.734 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:34:53.734 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.734 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:53.734 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.734 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:53.734 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:53.734 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:34:53.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.734 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.734 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.734 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:53.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.735 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.735 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:53.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.735 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.735 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:34:53.735 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:53.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.735 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.736 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:34:53.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:34:53.736 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.736 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.736 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.736 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:34:53.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:34:53.736 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.736 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.736 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.736 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:34:53.736 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:34:53.736 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.737 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.737 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:34:53.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:34:53.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.737 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.737 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:34:53.737 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:34:53.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.737 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:34:53.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:34:53.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:34:53.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:34:53.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.738 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.738 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.738 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:34:53.738 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:34:53.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.739 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.739 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:34:53.739 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:34:53.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.739 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.739 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.739 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:34:53.739 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:34:53.739 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.739 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.739 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.740 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.740 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:53.740 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:53.740 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.622814605000002, 2024-04-24T14:34:53.740 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:34:53.740 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.740 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:53.740 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.740 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:53.740 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:53.740 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:34:53.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.741 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.741 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:53.741 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.741 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:53.741 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.741 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.741 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:34:53.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:53.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.742 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:34:53.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:34:53.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.742 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.742 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.742 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:34:53.742 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:34:53.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:34:53.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:34:53.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:34:53.743 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:34:53.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.743 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.743 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.743 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:34:53.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:34:53.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.744 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.744 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:34:53.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:34:53.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.744 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.744 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:34:53.744 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:34:53.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:34:53.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:34:53.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:34:53.745 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:34:53.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.745 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.745 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.745 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:34:53.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:34:53.746 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.746 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.746 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.746 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.746 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:53.746 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:53.746 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.622775169999997, 2024-04-24T14:34:53.746 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:34:53.746 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.746 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:53.747 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.747 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:53.747 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:53.747 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:34:53.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.747 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.747 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:53.747 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.747 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:53.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:34:53.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:53.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.748 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.748 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.748 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:34:53.748 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:34:53.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.749 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.749 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:34:53.749 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:34:53.749 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.749 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.749 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:34:53.749 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:34:53.749 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.749 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.749 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.750 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:34:53.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:34:53.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.750 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.750 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.750 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:34:53.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:34:53.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.750 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.750 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.750 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:34:53.750 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:34:53.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:34:53.751 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:34:53.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.751 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.751 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:34:53.751 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:34:53.751 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.751 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.752 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.752 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:34:53.752 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:34:53.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.752 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.752 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.752 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:34:53.752 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:34:53.752 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.752 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.752 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.753 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.753 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:53.753 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:53.753 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.622750185999998, 2024-04-24T14:34:53.753 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:34:53.753 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.753 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:53.753 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.753 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:53.753 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:53.753 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:34:53.753 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.753 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.753 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.754 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.754 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:53.754 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.754 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.754 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.754 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.754 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:53.754 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.754 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.754 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.754 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.754 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:34:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:34:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:34:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:34:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:34:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.755 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.756 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:34:53.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:34:53.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.756 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.756 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.756 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:34:53.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:34:53.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.756 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.756 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.756 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:34:53.756 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:34:53.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.757 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:34:53.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:34:53.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.757 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:34:53.757 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:34:53.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.757 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.757 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.757 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:34:53.758 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:34:53.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.758 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.758 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.758 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:34:53.758 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:34:53.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.758 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.758 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.758 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:34:53.758 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:34:53.758 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.758 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.759 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.759 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.759 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.759 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:53.759 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:53.759 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 54.622724527000003, 2024-04-24T14:34:53.759 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:34:53.759 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:53.759 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:53.759 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:53.759 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:53.759 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:53.759 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:34:53.760 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.760 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:53.760 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.760 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:53.760 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:53.760 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.760 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.760 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.760 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:53.760 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:53.760 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:53.760 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.760 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.761 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:53.761 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:34:53.761 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:53.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.761 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.761 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:53.761 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:34:53.761 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:34:53.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.761 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.761 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:53.761 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:34:53.761 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:34:53.762 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.762 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.762 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:53.762 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:34:53.762 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:53.762 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.762 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.762 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.762 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:34:53.762 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:34:53.762 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.762 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.763 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:53.763 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:34:53.763 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:34:53.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.763 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.763 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.763 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:34:53.763 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:34:53.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.763 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.763 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:53.763 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:34:53.764 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:34:53.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.764 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.764 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:53.764 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:34:53.764 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:34:53.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.764 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.764 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:53.764 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:34:53.764 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:34:53.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:53.765 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:53.765 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:53.765 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:34:53.765 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:34:53.765 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.765 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.765 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.765 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:53.765 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:53.765 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:53.767 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:53.768 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:53.772+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:53.868 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:53.872+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:53.918 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:53.918 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:53.918 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:53.918 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:53.927 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:53.968 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:53.970+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:54.027 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:54.027 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:54.027 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:54.028 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:54.028 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:54.036 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:54.069 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:54.070+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:54.136 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:54.136 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:54.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:54.136 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:54.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.136 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:54.136 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:54.136 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.475049846000005, 2024-04-24T14:34:54.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:34:54.136 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.137 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:54.137 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.137 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:54.137 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:54.137 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:34:54.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.137 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:54.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:54.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:54.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:54.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:34:54.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:34:54.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:34:54.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:34:54.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:34:54.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:34:54.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:34:54.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:34:54.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:34:54.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:34:54.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:34:54.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:34:54.140 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.140 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.140 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.141 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:54.141 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:54.141 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.472015319999997, 2024-04-24T14:34:54.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:34:54.141 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.141 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:54.141 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.141 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:54.141 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:54.141 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:34:54.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.141 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:54.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:54.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:54.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:54.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:34:54.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:34:54.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:34:54.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:34:54.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:34:54.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:34:54.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:34:54.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:34:54.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:34:54.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:34:54.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.471991134000007, 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.145 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:54.146 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:54.146 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:34:54.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.146 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:54.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:54.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:54.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:54.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:34:54.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:34:54.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:34:54.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:34:54.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:34:54.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:34:54.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:34:54.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:34:54.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.471973750000004, 2024-04-24T14:34:54.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:34:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:34:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:54.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:34:54.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:54.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:34:54.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:34:54.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:34:54.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:34:54.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:34:54.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:34:54.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:34:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:34:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:34:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:34:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:34:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:34:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.153 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.154 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.154 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:54.154 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:54.154 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.457537658000007, 2024-04-24T14:34:54.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:34:54.154 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.154 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:54.154 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.154 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:54.154 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:54.154 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:34:54.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.155 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:54.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:54.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:34:54.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:54.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:34:54.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:34:54.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.156 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.156 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:34:54.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:34:54.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:34:54.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:34:54.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:34:54.158 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.158 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.158 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.158 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:54.158 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:54.158 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.457505358999995, 2024-04-24T14:34:54.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:34:54.159 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.159 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:54.159 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.159 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:54.159 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:54.159 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:34:54.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.159 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:54.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:54.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:34:54.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:54.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:34:54.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:34:54.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:34:54.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:34:54.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.162 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.162 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:34:54.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:34:54.162 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.162 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.162 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.163 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:54.163 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:54.163 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.443253081999998, 2024-04-24T14:34:54.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:34:54.163 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.163 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:54.163 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.163 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:54.163 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:54.163 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:34:54.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.163 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:34:54.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:34:54.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:34:54.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:34:54.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:34:54.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:34:54.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:34:54.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:34:54.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:34:54.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:34:54.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.443223982000006, 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.167 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:54.168 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:54.168 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:34:54.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.168 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:54.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.168 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.168 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:54.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:34:54.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:34:54.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:34:54.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:34:54.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:34:54.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:34:54.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:34:54.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:34:54.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:34:54.171 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.171 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.171 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.171 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:54.171 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:54.172 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.443198052, 2024-04-24T14:34:54.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:34:54.172 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.172 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:54.172 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.172 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:54.172 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:54.172 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:34:54.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.172 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:34:54.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:34:54.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:34:54.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:34:54.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.174 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.174 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:34:54.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:34:54.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:34:54.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:34:54.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:34:54.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:34:54.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:34:54.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:34:54.175 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.176 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.176 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.176 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:54.176 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:54.176 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.427477588000002, 2024-04-24T14:34:54.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:34:54.176 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.176 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:54.176 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.176 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:54.176 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:54.176 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:34:54.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.177 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:54.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:54.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:34:54.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:54.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:34:54.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:34:54.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:34:54.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:34:54.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:34:54.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:34:54.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:34:54.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:34:54.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:34:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:34:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:34:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:34:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:54.180 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:54.181 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.427445656000003, 2024-04-24T14:34:54.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:34:54.181 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.181 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:54.181 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.181 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:54.181 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:54.181 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:34:54.181 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.181 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.181 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.181 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.181 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:54.181 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:54.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.182 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:34:54.182 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:54.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.182 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.182 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.183 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:54.184 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.427378140000002, 2024-04-24T14:34:54.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:34:54.185 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.185 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:54.185 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.185 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:54.185 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:54.185 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:34:54.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.185 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.185 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.185 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.185 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:54.185 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.185 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:54.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:34:54.186 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:54.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.186 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.186 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.186 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:34:54.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:34:54.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:34:54.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:34:54.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.187 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.187 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:34:54.187 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:34:54.187 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.187 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:34:54.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:34:54.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.188 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.188 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:34:54.188 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:34:54.188 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.188 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.188 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.188 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.188 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:34:54.189 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:54.189 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.369965257000004, 2024-04-24T14:34:54.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:34:54.189 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.189 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.189 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.189 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:54.189 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.189 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.189 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.189 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:54.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.190 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:54.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:54.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.190 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.190 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:34:54.190 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:34:54.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.190 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.190 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:34:54.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:34:54.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:34:54.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:34:54.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.191 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.191 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:34:54.191 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:34:54.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.191 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:34:54.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:34:54.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:34:54.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:34:54.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.192 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.192 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.192 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:34:54.192 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:34:54.193 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.193 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.193 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.193 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:103 3.1 e87/74 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head v 87'261, mlcod=87'258)", 2024-04-24T14:34:54.193 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:54.193 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.178684053000001, 2024-04-24T14:34:54.193 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019745728000000001, 2024-04-24T14:34:54.193 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.193 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.193 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.193 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:54.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:54.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.194 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.194 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.194 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960679+0000", 2024-04-24T14:34:54.194 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:54.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960692+0000", 2024-04-24T14:34:54.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3709999999999999e-05 2024-04-24T14:34:54.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960722+0000", 2024-04-24T14:34:54.195 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9791999999999999e-05 2024-04-24T14:34:54.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.195 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.195 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.195 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960733+0000", 2024-04-24T14:34:54.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.128e-05 2024-04-24T14:34:54.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960857+0000", 2024-04-24T14:34:54.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000123981 2024-04-24T14:34:54.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.196 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960926+0000", 2024-04-24T14:34:54.196 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8881999999999995e-05 2024-04-24T14:34:54.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.196 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980353+0000", 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019426853000000001 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980426+0000", 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2803000000000001e-05 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:105 3.1 e87/74 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head v 87'263, mlcod=87'258)", 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:54.197 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.169557519000001, 2024-04-24T14:34:54.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014864743999999999, 2024-04-24T14:34:54.198 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.198 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.198 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:54.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.198 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.198 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.198 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:54.198 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969805+0000", 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969820+0000", 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5116999999999999e-05 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969852+0000", 2024-04-24T14:34:54.199 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2135999999999999e-05 2024-04-24T14:34:54.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.200 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969864+0000", 2024-04-24T14:34:54.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.189e-05 2024-04-24T14:34:54.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.200 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972122+0000", 2024-04-24T14:34:54.200 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022580389999999999 2024-04-24T14:34:54.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.200 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.200 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.200 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972165+0000", 2024-04-24T14:34:54.201 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2429000000000003e-05 2024-04-24T14:34:54.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.201 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.201 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984619+0000", 2024-04-24T14:34:54.201 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012454735 2024-04-24T14:34:54.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.201 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.201 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.201 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984672+0000", 2024-04-24T14:34:54.201 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2077999999999998e-05 2024-04-24T14:34:54.201 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.201 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.202 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.202 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.202 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:112 3.1 e87/74 3:a983798d:::rbd_data.12ff64619e2d.0000000000000006:head v 87'265, mlcod=87'258)", 2024-04-24T14:34:54.202 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:54.202 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.159218682999999, 2024-04-24T14:34:54.202 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012821978, 2024-04-24T14:34:54.202 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.202 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.202 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.202 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.202 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.202 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:54.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:54.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.203 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980144+0000", 2024-04-24T14:34:54.203 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:54.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.203 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.203 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980159+0000", 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4769999999999999e-05 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980189+0000", 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0419000000000001e-05 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980213+0000", 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4036000000000002e-05 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.204 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980452+0000", 2024-04-24T14:34:54.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00023921500000000001 2024-04-24T14:34:54.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.205 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980468+0000", 2024-04-24T14:34:54.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5843999999999999e-05 2024-04-24T14:34:54.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.205 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992932+0000", 2024-04-24T14:34:54.205 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012463314 2024-04-24T14:34:54.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.205 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.205 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.206 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992968+0000", 2024-04-24T14:34:54.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6075999999999998e-05 2024-04-24T14:34:54.206 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.206 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.206 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.206 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.206 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.206 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:113 3.1 e87/74 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head v 87'267, mlcod=87'258)", 2024-04-24T14:34:54.206 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:54.206 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.155683959000001, 2024-04-24T14:34:54.206 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010561022999999999, 2024-04-24T14:34:54.206 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.206 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.207 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:54.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:54.207 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.207 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.207 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.207 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983679+0000", 2024-04-24T14:34:54.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:54.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983699+0000", 2024-04-24T14:34:54.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0392000000000001e-05 2024-04-24T14:34:54.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.208 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.208 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.208 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983737+0000", 2024-04-24T14:34:54.208 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8340000000000002e-05 2024-04-24T14:34:54.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983756+0000", 2024-04-24T14:34:54.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8402000000000001e-05 2024-04-24T14:34:54.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984706+0000", 2024-04-24T14:34:54.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095024400000000005 2024-04-24T14:34:54.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.209 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.209 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984741+0000", 2024-04-24T14:34:54.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4721000000000002e-05 2024-04-24T14:34:54.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.210 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994211+0000", 2024-04-24T14:34:54.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094708559999999997 2024-04-24T14:34:54.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.210 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.210 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994241+0000", 2024-04-24T14:34:54.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9941e-05 2024-04-24T14:34:54.210 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.210 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.210 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.211 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:115 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:54.211 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:54.211 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.155229922, 2024-04-24T14:34:54.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010347242, 2024-04-24T14:34:54.211 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.211 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.211 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.211 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:34:54.211 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:54.211 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:54.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.211 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984132+0000", 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984142+0000", 2024-04-24T14:34:54.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7370000000000001e-06 2024-04-24T14:34:54.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984154+0000", 2024-04-24T14:34:54.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2288e-05 2024-04-24T14:34:54.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984163+0000", 2024-04-24T14:34:54.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3749999999999992e-06 2024-04-24T14:34:54.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.214 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984246+0000", 2024-04-24T14:34:54.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2962999999999998e-05 2024-04-24T14:34:54.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.214 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984473+0000", 2024-04-24T14:34:54.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00022696399999999999 2024-04-24T14:34:54.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:54.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984771+0000", 2024-04-24T14:34:54.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00029739499999999999 2024-04-24T14:34:54.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:54.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992998+0000", 2024-04-24T14:34:54.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082275750000000009 2024-04-24T14:34:54.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:54.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994299+0000", 2024-04-24T14:34:54.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013005289999999999 2024-04-24T14:34:54.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994403+0000", 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010376499999999999 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994482+0000", 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9013999999999999e-05 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:128 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:54.216 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.134038994000001, 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011784144, 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 128 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:54.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005324+0000", 2024-04-24T14:34:54.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:54.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005332+0000", 2024-04-24T14:34:54.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2989999999999997e-06 2024-04-24T14:34:54.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005354+0000", 2024-04-24T14:34:54.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1705999999999999e-05 2024-04-24T14:34:54.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005362+0000", 2024-04-24T14:34:54.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5599999999999996e-06 2024-04-24T14:34:54.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005418+0000", 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6685999999999999e-05 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005535+0000", 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117008 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005732+0000", 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019718700000000001 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:54.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016440+0000", 2024-04-24T14:34:54.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010707787999999999 2024-04-24T14:34:54.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:54.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016913+0000", 2024-04-24T14:34:54.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047270400000000001 2024-04-24T14:34:54.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.221 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.221 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017047+0000", 2024-04-24T14:34:54.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013434699999999999 2024-04-24T14:34:54.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017109+0000", 2024-04-24T14:34:54.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2192000000000004e-05 2024-04-24T14:34:54.222 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.222 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.222 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.222 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:130 3.1 e87/74 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 87'269, mlcod=87'265)", 2024-04-24T14:34:54.222 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:54.222 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 27.133586233999999, 2024-04-24T14:34:54.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013945631999999999, 2024-04-24T14:34:54.222 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.223 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.223 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:54.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:54.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005777+0000", 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005791+0000", 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3743999999999999e-05 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005803+0000", 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2330999999999999e-05 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005810+0000", 2024-04-24T14:34:54.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8990000000000004e-06 2024-04-24T14:34:54.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005888+0000", 2024-04-24T14:34:54.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7872000000000003e-05 2024-04-24T14:34:54.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005911+0000", 2024-04-24T14:34:54.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3156e-05 2024-04-24T14:34:54.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019642+0000", 2024-04-24T14:34:54.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013730456 2024-04-24T14:34:54.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019724+0000", 2024-04-24T14:34:54.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1981e-05 2024-04-24T14:34:54.226 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.226 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.226 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.226 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.226 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.227 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:54.228 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:54.228 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:54.234+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:54.329 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:54.332+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:54.331 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:54.331 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:54.331 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:54.340 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:54.430 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:54.434+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:54.438 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:54.438 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:54.438 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:54.438 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:54.438 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:54.447 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:54.531 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:54.534+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:54.546 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:54.546 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:54.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:54.546 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:54.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.547 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:34:54.547 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:54.547 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.477971280999995, 2024-04-24T14:34:54.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:34:54.547 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.547 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.547 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:54.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:54.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:34:54.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:54.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:34:54.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:34:54.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:34:54.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:34:54.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:34:54.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:34:54.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:34:54.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:34:54.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:34:54.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:34:54.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:34:54.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:34:54.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:34:54.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:34:54.552 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.552 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.552 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.552 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:34:54.552 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:54.552 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 60.483156702000002, 2024-04-24T14:34:54.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:34:54.552 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.552 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.552 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:54.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:54.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:34:54.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:54.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:34:54.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:34:54.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:34:54.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:34:54.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:34:54.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:34:54.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:34:54.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:34:54.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:34:54.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:34:54.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:34:54.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:34:54.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:34:54.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:34:54.557 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.557 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.557 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.558 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:34:54.558 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:54.558 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 60.472227443999998, 2024-04-24T14:34:54.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:34:54.558 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.558 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.558 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:54.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:54.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:34:54.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:54.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:34:54.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:34:54.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:34:54.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:34:54.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:34:54.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:34:54.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:34:54.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:34:54.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:34:54.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:34:54.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:34:54.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:34:54.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:34:54.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:34:54.562 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.562 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.562 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.563 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:34:54.563 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:54.563 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 60.461271363000002, 2024-04-24T14:34:54.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:34:54.563 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.563 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.563 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:54.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:54.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:34:54.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:54.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:34:54.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:34:54.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:34:54.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:34:54.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:34:54.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:34:54.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:34:54.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:34:54.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:34:54.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:34:54.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:34:54.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:34:54.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:34:54.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:34:54.567 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.568 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.568 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.568 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:34:54.568 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:54.568 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 60.449701998000002, 2024-04-24T14:34:54.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:34:54.568 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.568 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.568 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:54.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:54.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:34:54.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:54.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:34:54.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:34:54.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:34:54.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:34:54.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:34:54.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:34:54.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:34:54.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:34:54.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:34:54.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:34:54.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:34:54.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:34:54.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:34:54.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:34:54.573 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.573 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.573 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.573 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:34:54.573 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:54.573 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 60.196582339000003, 2024-04-24T14:34:54.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:34:54.573 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.574 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.574 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:54.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:54.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:34:54.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:54.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:34:54.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:34:54.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:34:54.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:34:54.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:34:54.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:34:54.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:34:54.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:34:54.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:34:54.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:34:54.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:34:54.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:34:54.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:34:54.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:34:54.578 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.578 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.578 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.579 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:34:54.579 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:54.579 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 60.185837171000003, 2024-04-24T14:34:54.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:34:54.579 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.579 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.579 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:54.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:54.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:34:54.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:54.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:34:54.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:34:54.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:34:54.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:34:54.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:34:54.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:34:54.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:34:54.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:34:54.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:34:54.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:34:54.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:34:54.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:34:54.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:34:54.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:34:54.583 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.583 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.584 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.584 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:34:54.584 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:54.584 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 60.173470895000001, 2024-04-24T14:34:54.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:34:54.584 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.584 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.584 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:54.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:54.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:34:54.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:54.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:34:54.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:34:54.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:34:54.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:34:54.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:34:54.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:34:54.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:34:54.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:34:54.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:34:54.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:34:54.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:34:54.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:34:54.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:34:54.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:34:54.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.589 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.589 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:34:54.589 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:54.589 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 60.048534773999997, 2024-04-24T14:34:54.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:34:54.590 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.590 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.590 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:54.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:54.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:34:54.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:54.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:34:54.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:34:54.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:34:54.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:34:54.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:34:54.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:34:54.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:34:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:34:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:34:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:34:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:34:54.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:34:54.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:34:54.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:34:54.594 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.594 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.594 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.594 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:34:54.595 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:54.595 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 60.029146931, 2024-04-24T14:34:54.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:34:54.595 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.595 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.595 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:54.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:54.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:34:54.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:54.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:34:54.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:34:54.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:34:54.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:34:54.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:34:54.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:34:54.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:34:54.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:34:54.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:34:54.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:34:54.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:34:54.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:34:54.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:34:54.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:34:54.599 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.599 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.600 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.600 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:34:54.600 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:54.600 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 60.019419087999999, 2024-04-24T14:34:54.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:34:54.600 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.600 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.600 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:54.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:54.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:34:54.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:54.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:34:54.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:34:54.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:34:54.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:34:54.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:34:54.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:34:54.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:34:54.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:34:54.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:34:54.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:34:54.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:34:54.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:34:54.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:34:54.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:34:54.605 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.605 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.605 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.605 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:34:54.605 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:54.606 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.859309408000001, 2024-04-24T14:34:54.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:34:54.606 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.606 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.606 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:54.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:54.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:34:54.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:54.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:34:54.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:34:54.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:34:54.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:34:54.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:34:54.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:34:54.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:34:54.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:34:54.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:34:54.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:34:54.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:34:54.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:34:54.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:34:54.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:34:54.610 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.610 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.611 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.611 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:34:54.611 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:54.611 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.849580412000002, 2024-04-24T14:34:54.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:34:54.611 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.611 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.611 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:54.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:54.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:34:54.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:54.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:34:54.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:34:54.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:34:54.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:34:54.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:34:54.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:34:54.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:34:54.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:34:54.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:34:54.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:34:54.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:34:54.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:34:54.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:34:54.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:34:54.616 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.616 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.616 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.616 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:34:54.616 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:54.616 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 55.514609577000002, 2024-04-24T14:34:54.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:34:54.617 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.617 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:54.617 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:54.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:54.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:54.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:54.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:34:54.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:34:54.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:34:54.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:34:54.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:34:54.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:34:54.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:34:54.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:34:54.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:34:54.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:34:54.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:54.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:34:54.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:34:54.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:34:54.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:34:54.621 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.621 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.621 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.622 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:54.622 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:54.622 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.670458828999998, 2024-04-24T14:34:54.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:34:54.622 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.622 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:54.622 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:54.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:54.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:54.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:54.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:34:54.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:34:54.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:34:54.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:34:54.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:34:54.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:34:54.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:34:54.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:34:54.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:34:54.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:34:54.626 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.626 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.626 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.626 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:5 3.2 3.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:34:54.626 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:54.626 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.342221187, 2024-04-24T14:34:54.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043717790000000001, 2024-04-24T14:34:54.627 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.627 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-24T14:34:54.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:54.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:34:54.627 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:34:54.627 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 5 2024-04-24T14:34:54.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.627 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:54.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:54.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207678+0000", 2024-04-24T14:34:54.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999962 2024-04-24T14:34:54.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207700+0000", 2024-04-24T14:34:54.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1313000000000001e-05 2024-04-24T14:34:54.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207757+0000", 2024-04-24T14:34:54.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7499e-05 2024-04-24T14:34:54.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207779+0000", 2024-04-24T14:34:54.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1478999999999999e-05 2024-04-24T14:34:54.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.212054+0000", 2024-04-24T14:34:54.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042750849999999996 2024-04-24T14:34:54.630 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.630 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.630 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.631 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:54.631 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:54.631 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.446695828999999, 2024-04-24T14:34:54.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044084859999999997, 2024-04-24T14:34:54.631 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.631 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:54.631 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:54.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:54.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103206+0000", 2024-04-24T14:34:54.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:54.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104731+0000", 2024-04-24T14:34:54.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001524755 2024-04-24T14:34:54.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104759+0000", 2024-04-24T14:34:54.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7973000000000001e-05 2024-04-24T14:34:54.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104807+0000", 2024-04-24T14:34:54.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8112000000000003e-05 2024-04-24T14:34:54.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104907+0000", 2024-04-24T14:34:54.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100317 2024-04-24T14:34:54.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104918+0000", 2024-04-24T14:34:54.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747999999999999e-05 2024-04-24T14:34:54.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.107616+0000", 2024-04-24T14:34:54.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026977559999999999 2024-04-24T14:34:54.635 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.635 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.636 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.636 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(3.2 83/82 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:54.636 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:54.636 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.440301452, 2024-04-24T14:34:54.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041425760000000002, 2024-04-24T14:34:54.636 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.636 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:54.636 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:54.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:54.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109600+0000", 2024-04-24T14:34:54.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:54.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111210+0000", 2024-04-24T14:34:54.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016097279999999999 2024-04-24T14:34:54.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111237+0000", 2024-04-24T14:34:54.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7189999999999999e-05 2024-04-24T14:34:54.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111251+0000", 2024-04-24T14:34:54.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.365e-05 2024-04-24T14:34:54.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111367+0000", 2024-04-24T14:34:54.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000115703 2024-04-24T14:34:54.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111378+0000", 2024-04-24T14:34:54.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1873e-05 2024-04-24T14:34:54.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.113744+0000", 2024-04-24T14:34:54.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023657069999999999 2024-04-24T14:34:54.640 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.640 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.641 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.641 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:54.641 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:54.641 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.423400876000002, 2024-04-24T14:34:54.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046193859999999996, 2024-04-24T14:34:54.641 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.641 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:54.641 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:54.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:54.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126501+0000", 2024-04-24T14:34:54.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:54.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128169+0000", 2024-04-24T14:34:54.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016686229999999999 2024-04-24T14:34:54.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128202+0000", 2024-04-24T14:34:54.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2539000000000001e-05 2024-04-24T14:34:54.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128222+0000", 2024-04-24T14:34:54.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0149e-05 2024-04-24T14:34:54.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128375+0000", 2024-04-24T14:34:54.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000153459 2024-04-24T14:34:54.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128399+0000", 2024-04-24T14:34:54.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3393000000000001e-05 2024-04-24T14:34:54.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.131122+0000", 2024-04-24T14:34:54.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027227140000000002 2024-04-24T14:34:54.645 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.645 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.645 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.646 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 49'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@49'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:54.646 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:54.646 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.401619496000002, 2024-04-24T14:34:54.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038414529999999999, 2024-04-24T14:34:54.646 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:54.646 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:54.646 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:54.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:54.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:54.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:54.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:54.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:54.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:54.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148282+0000", 2024-04-24T14:34:54.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:54.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:54.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149875+0000", 2024-04-24T14:34:54.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015929900000000001 2024-04-24T14:34:54.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:54.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149908+0000", 2024-04-24T14:34:54.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3102000000000003e-05 2024-04-24T14:34:54.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149941+0000", 2024-04-24T14:34:54.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2480000000000001e-05 2024-04-24T14:34:54.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:54.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150031+0000", 2024-04-24T14:34:54.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0394000000000002e-05 2024-04-24T14:34:54.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:54.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150043+0000", 2024-04-24T14:34:54.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1967000000000001e-05 2024-04-24T14:34:54.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:54.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:54.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:54.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.152125+0000", 2024-04-24T14:34:54.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020817029999999999 2024-04-24T14:34:54.650 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.651 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.651 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.651 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:54.651 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:54.651 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:54.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:54.653 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:54.654+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:54.754 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:54.756+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:54.759 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:54.759 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:54.760 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:54.760 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:54.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:54.855 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:54.856+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:54.870 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:54.870 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:54.870 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:54.871 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:54.871 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:54.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:54.955 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:54.958+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:54.980 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:54.980 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:54.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:54.980 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:54.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.105560138999998, 2024-04-24T14:34:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:34:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:34:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:34:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:34:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:54.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:34:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:34:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:34:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:34:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:54.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:54.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:34:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:34:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:34:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:34:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.986 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.105531153999998, 2024-04-24T14:34:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:34:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:34:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:54.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:54.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:34:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:34:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:34:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:34:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:34:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:34:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:34:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:34:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:34:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:34:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:54.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:34:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:34:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:34:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:34:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:34:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:34:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:54.992 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.105512668999999, 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:54.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:34:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:34:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:34:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:34:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:34:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:34:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:34:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:34:54.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:34:54.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:34:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:34:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:34:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:34:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:34:54.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:34:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:34:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:34:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:54.998 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.105497388000003, 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:54.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:34:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:34:55.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:34:55.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:34:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:34:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:34:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:34:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:34:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:55.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:34:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:34:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:34:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:34:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:34:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:34:55.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:34:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:34:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.105481607000002, 2024-04-24T14:34:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:34:55.004 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:34:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:34:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:34:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:34:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:34:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:34:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:34:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:34:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:34:55.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.009 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.105467318999999, 2024-04-24T14:34:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:34:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:55.010 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:34:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:34:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:34:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:34:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:34:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:34:55.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:34:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:34:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:34:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:34:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:34:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.105447566999999, 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:34:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:34:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:34:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:34:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:34:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:34:55.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:34:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:34:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:34:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:34:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:34:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.105430120000001, 2024-04-24T14:34:55.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:34:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:34:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:34:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:34:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:55.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:34:55.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:34:55.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:34:55.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:34:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:34:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:34:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:34:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:34:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:34:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:100 3.2 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.231375305, 2024-04-24T14:34:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009078015, 2024-04-24T14:34:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:55.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 100 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754019+0000", 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754025+0000", 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2110000000000003e-06 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754037+0000", 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1868999999999999e-05 2024-04-24T14:34:55.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754046+0000", 2024-04-24T14:34:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7430000000000008e-06 2024-04-24T14:34:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754095+0000", 2024-04-24T14:34:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8924000000000001e-05 2024-04-24T14:34:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754153+0000", 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8165000000000003e-05 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754262+0000", 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108694 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.762999+0000", 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087372590000000007 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:55.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763027+0000", 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8467e-05 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763070+0000", 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2236e-05 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763098+0000", 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8561999999999999e-05 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:101 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [delete] snapc 7=[7,6,4] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.026167639000001, 2024-04-24T14:34:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012894743, 2024-04-24T14:34:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 101 2024-04-24T14:34:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:34:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959227+0000", 2024-04-24T14:34:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959233+0000", 2024-04-24T14:34:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.857e-06 2024-04-24T14:34:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959249+0000", 2024-04-24T14:34:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6155999999999999e-05 2024-04-24T14:34:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959257+0000", 2024-04-24T14:34:55.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7440000000000007e-06 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959790+0000", 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053381100000000003 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960206+0000", 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041549700000000002 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960744+0000", 2024-04-24T14:34:55.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053762199999999997 2024-04-24T14:34:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971963+0000", 2024-04-24T14:34:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011219816000000001 2024-04-24T14:34:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972021+0000", 2024-04-24T14:34:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7855000000000001e-05 2024-04-24T14:34:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972073+0000", 2024-04-24T14:34:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2196e-05 2024-04-24T14:34:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972123+0000", 2024-04-24T14:34:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9311999999999998e-05 2024-04-24T14:34:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:102 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:55.038 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.026127613, 2024-04-24T14:34:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012939713, 2024-04-24T14:34:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 102 2024-04-24T14:34:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:55.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959267+0000", 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959274+0000", 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.593e-06 2024-04-24T14:34:55.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959291+0000", 2024-04-24T14:34:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6387000000000001e-05 2024-04-24T14:34:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959301+0000", 2024-04-24T14:34:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0023e-05 2024-04-24T14:34:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969594+0000", 2024-04-24T14:34:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010293182 2024-04-24T14:34:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969902+0000", 2024-04-24T14:34:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030835299999999998 2024-04-24T14:34:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970408+0000", 2024-04-24T14:34:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050588100000000002 2024-04-24T14:34:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972009+0000", 2024-04-24T14:34:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016011210000000001 2024-04-24T14:34:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972132+0000", 2024-04-24T14:34:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122521 2024-04-24T14:34:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972174+0000", 2024-04-24T14:34:55.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2242e-05 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972208+0000", 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3522999999999997e-05 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:104 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.025516399000001, 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014589062, 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.044 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 104 2024-04-24T14:34:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:34:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959877+0000", 2024-04-24T14:34:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959890+0000", 2024-04-24T14:34:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2537e-05 2024-04-24T14:34:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959909+0000", 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9340000000000001e-05 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959922+0000", 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2882e-05 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970900+0000", 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010977736 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971039+0000", 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139593 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971453+0000", 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041351099999999998 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974044+0000", 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025908289999999998 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974377+0000", 2024-04-24T14:34:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033310399999999997 2024-04-24T14:34:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974433+0000", 2024-04-24T14:34:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6166999999999999e-05 2024-04-24T14:34:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974468+0000", 2024-04-24T14:34:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5336000000000002e-05 2024-04-24T14:34:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:106 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.025457066000001, 2024-04-24T14:34:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017192064999999999, 2024-04-24T14:34:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:55.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 106 2024-04-24T14:34:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:34:55.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959938+0000", 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959943+0000", 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.677e-06 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959953+0000", 2024-04-24T14:34:55.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8069999999999994e-06 2024-04-24T14:34:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:34:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0560000000000001e-06 2024-04-24T14:34:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972232+0000", 2024-04-24T14:34:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012272460000000001 2024-04-24T14:34:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972304+0000", 2024-04-24T14:34:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2630000000000001e-05 2024-04-24T14:34:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972548+0000", 2024-04-24T14:34:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000244047 2024-04-24T14:34:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976900+0000", 2024-04-24T14:34:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043514069999999998 2024-04-24T14:34:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977076+0000", 2024-04-24T14:34:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176281 2024-04-24T14:34:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977106+0000", 2024-04-24T14:34:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0015e-05 2024-04-24T14:34:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977131+0000", 2024-04-24T14:34:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4669e-05 2024-04-24T14:34:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:114 3.2 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.002135901999999, 2024-04-24T14:34:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090880089999999993, 2024-04-24T14:34:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 114 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:34:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983259+0000", 2024-04-24T14:34:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983264+0000", 2024-04-24T14:34:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9130000000000002e-06 2024-04-24T14:34:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983277+0000", 2024-04-24T14:34:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3723999999999999e-05 2024-04-24T14:34:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983282+0000", 2024-04-24T14:34:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0429999999999998e-06 2024-04-24T14:34:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983339+0000", 2024-04-24T14:34:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6323000000000003e-05 2024-04-24T14:34:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983464+0000", 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012554299999999999 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983693+0000", 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000228848 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.986978+0000", 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032852810000000001 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992288+0000", 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053095369999999996 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992322+0000", 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3683999999999998e-05 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992348+0000", 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6157999999999999e-05 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.061 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:121 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.990079803, 2024-04-24T14:34:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098798159999999996, 2024-04-24T14:34:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:55.062 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 121 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995315+0000", 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995320+0000", 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0239999999999997e-06 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995328+0000", 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7800000000000001e-06 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995334+0000", 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2330000000000003e-06 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998810+0000", 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034756090000000002 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998870+0000", 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0625999999999999e-05 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:55.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.999091+0000", 2024-04-24T14:34:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000220631 2024-04-24T14:34:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.001389+0000", 2024-04-24T14:34:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022977549999999998 2024-04-24T14:34:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005097+0000", 2024-04-24T14:34:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037085439999999998 2024-04-24T14:34:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005154+0000", 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6208000000000001e-05 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005196+0000", 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2052000000000003e-05 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:127 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:55.067 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.979778286999998, 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013414918, 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005616+0000", 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005622+0000", 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2020000000000003e-06 2024-04-24T14:34:55.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005632+0000", 2024-04-24T14:34:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2024-04-24T14:34:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005637+0000", 2024-04-24T14:34:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7439999999999998e-06 2024-04-24T14:34:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005947+0000", 2024-04-24T14:34:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000310792 2024-04-24T14:34:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006059+0000", 2024-04-24T14:34:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011192399999999999 2024-04-24T14:34:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006503+0000", 2024-04-24T14:34:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000443394 2024-04-24T14:34:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018864+0000", 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012361162 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018877+0000", 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3497e-05 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018965+0000", 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7476000000000001e-05 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019032+0000", 2024-04-24T14:34:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7469000000000001e-05 2024-04-24T14:34:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:129 3.2 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.979638053999999, 2024-04-24T14:34:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010059754000000001, 2024-04-24T14:34:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:55.073 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 129 2024-04-24T14:34:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005757+0000", 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005760+0000", 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2430000000000001e-06 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005769+0000", 2024-04-24T14:34:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4810000000000006e-06 2024-04-24T14:34:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005773+0000", 2024-04-24T14:34:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005840+0000", 2024-04-24T14:34:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6934999999999995e-05 2024-04-24T14:34:55.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005961+0000", 2024-04-24T14:34:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012153 2024-04-24T14:34:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:34:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006181+0000", 2024-04-24T14:34:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021947500000000001 2024-04-24T14:34:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015520+0000", 2024-04-24T14:34:55.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093393049999999991 2024-04-24T14:34:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015733+0000", 2024-04-24T14:34:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021355699999999999 2024-04-24T14:34:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015773+0000", 2024-04-24T14:34:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9554000000000001e-05 2024-04-24T14:34:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015817+0000", 2024-04-24T14:34:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4325e-05 2024-04-24T14:34:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:131 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.979360173, 2024-04-24T14:34:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014251976, 2024-04-24T14:34:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:55.079 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131 2024-04-24T14:34:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:34:55.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006034+0000", 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006042+0000", 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2970000000000004e-06 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006061+0000", 2024-04-24T14:34:55.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8505999999999999e-05 2024-04-24T14:34:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006071+0000", 2024-04-24T14:34:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0409e-05 2024-04-24T14:34:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007899+0000", 2024-04-24T14:34:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018274789999999999 2024-04-24T14:34:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008281+0000", 2024-04-24T14:34:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038244700000000001 2024-04-24T14:34:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008528+0000", 2024-04-24T14:34:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024723900000000001 2024-04-24T14:34:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019070+0000", 2024-04-24T14:34:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010541867999999999 2024-04-24T14:34:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020099+0000", 2024-04-24T14:34:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00102852 2024-04-24T14:34:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020226+0000", 2024-04-24T14:34:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012744300000000001 2024-04-24T14:34:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020287+0000", 2024-04-24T14:34:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1253000000000005e-05 2024-04-24T14:34:55.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:132 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.979306319999999, 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015917981000000001, 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 132 2024-04-24T14:34:55.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006093+0000", 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3819999999999997e-06 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006103+0000", 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0352e-05 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006110+0000", 2024-04-24T14:34:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3910000000000003e-06 2024-04-24T14:34:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015586+0000", 2024-04-24T14:34:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094759119999999995 2024-04-24T14:34:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015730+0000", 2024-04-24T14:34:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014487199999999999 2024-04-24T14:34:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015886+0000", 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015573799999999999 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020306+0000", 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044199399999999998 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021887+0000", 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015809400000000001 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021942+0000", 2024-04-24T14:34:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5071000000000002e-05 2024-04-24T14:34:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.022007+0000", 2024-04-24T14:34:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5110000000000005e-05 2024-04-24T14:34:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.090 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.5324.0:127 3.0 e87/69 ondisk, result = 0)", 2024-04-24T14:34:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.977597251999999, 2024-04-24T14:34:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011245125, 2024-04-24T14:34:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:34:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007804+0000", 2024-04-24T14:34:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6610000000000003e-06 2024-04-24T14:34:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007817+0000", 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3030999999999999e-05 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007832+0000", 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.47e-05 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018843+0000", 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011010647 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018853+0000", 2024-04-24T14:34:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319000000000001e-05 2024-04-24T14:34:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019043+0000", 2024-04-24T14:34:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019055600000000001 2024-04-24T14:34:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:55.094 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:55.094 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:55.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:34:55.096 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:55.102+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:55.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:55.197 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:55.198+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:55.243 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:55.243 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:55.243 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:55.253 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:55.274 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:55.294 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:55.298 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:55.300+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:55.314 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:55.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T14:34:55.340 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:55.345 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:55.345 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:55.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:55.363 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:55.385 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:55.399 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:55.404+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:55.459 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:55.459 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:55.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:55.500 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:55.504+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:55.574 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:55.574 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:55.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:55.601 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:55.604+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:55.679 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:55.679 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:55.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:55.703 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:55.706+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:55.786 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:55.786 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:55.795 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:55.839 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:55.844+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:55.905 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:55.908+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:55.906 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:55.916 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:56.006 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:56.010+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:56.020 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:56.021 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:56.030 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:56.107 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:56.110+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:56.129 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:56.129 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:56.139 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:56.207 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:56.210+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:56.242 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:56.251 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:56.266 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-24T14:34:56.308 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:56.312+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:56.348 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:56.349 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:56.358 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:56.408 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:56.412+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:56.500 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:56.500 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:56.510 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:56.511 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:56.512+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:56.610 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:56.612+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:56.655 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:56.665 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:56.670 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4] out_osds: [5, 1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 1] 2024-04-24T14:34:56.670 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:34:56.670 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-24T14:34:56.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-04-24T14:34:56.711 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:56.714+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:56.767 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:56.777 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:56.812 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:56.816+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:56.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-04-24T14:34:56.871 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:56.871 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:56.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:56.926 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:56.930+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:56.985 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:56.985 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:56.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-04-24T14:34:56.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:57.014 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:57.018+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:57.084 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:57.085 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:57.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:57.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-04-24T14:34:57.124 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:57.126+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:57.216 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:57.220+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:57.267 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:57.268 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:57.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:34:57.316 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:57.320+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:57.364 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:57.364 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:57.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:34:57.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:34:57.417 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:57.420+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:57.490 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:57.490 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:57.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:34:57.518 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:57.522+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:57.541 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:57.562 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:57.583 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:57.603 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:57.603 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:57.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:57.603 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:57.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.603 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:57.603 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:57.603 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.851586669, 2024-04-24T14:34:57.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:34:57.603 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.603 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:57.604 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.604 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:57.604 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:57.604 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:34:57.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.604 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:57.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:34:57.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:34:57.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:57.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:34:57.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:34:57.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:34:57.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:34:57.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:34:57.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:34:57.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:34:57.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:34:57.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:57.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:34:57.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:34:57.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:34:57.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:34:57.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:57.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:34:57.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:34:57.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:34:57.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:34:57.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:34:57.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:34:57.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:34:57.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:34:57.609 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.609 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.609 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.609 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:57.609 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:57.609 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.851535834999993, 2024-04-24T14:34:57.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:34:57.609 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.609 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:57.610 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.610 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:57.610 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:57.610 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:34:57.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.610 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:57.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:34:57.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:34:57.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:34:57.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:34:57.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:34:57.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:34:57.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:34:57.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:34:57.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:34:57.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:34:57.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:34:57.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:34:57.615 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.615 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.615 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.615 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:57.615 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:57.615 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.851499528000005, 2024-04-24T14:34:57.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:34:57.615 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.615 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:57.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:57.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:57.616 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:34:57.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.616 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:57.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:34:57.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:34:57.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:57.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:34:57.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:34:57.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:34:57.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:34:57.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:34:57.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:34:57.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:34:57.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:34:57.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:57.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:34:57.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:34:57.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:34:57.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:34:57.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:57.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:34:57.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:34:57.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:34:57.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:34:57.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:34:57.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:34:57.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:34:57.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:34:57.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.621 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.621 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:57.621 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:57.621 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.848088278000006, 2024-04-24T14:34:57.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:34:57.622 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.622 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:57.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:57.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:57.622 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:34:57.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.622 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:57.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:34:57.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:34:57.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:34:57.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:34:57.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:34:57.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:34:57.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:34:57.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:57.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:34:57.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:34:57.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:34:57.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:34:57.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:57.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:34:57.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:34:57.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:34:57.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:34:57.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:34:57.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:34:57.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:34:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:34:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.848026601000001, 2024-04-24T14:34:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:34:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:57.628 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.628 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:57.628 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:57.628 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:34:57.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.628 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:57.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:34:57.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:34:57.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:57.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:34:57.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:34:57.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:34:57.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:34:57.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:34:57.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:34:57.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:34:57.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:34:57.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:57.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:34:57.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:34:57.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:34:57.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:34:57.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:57.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:34:57.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:34:57.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:34:57.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:34:57.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:34:57.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:34:57.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:34:57.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:34:57.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.633 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.633 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:57.633 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:57.633 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.847992024999996, 2024-04-24T14:34:57.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:34:57.633 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.633 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:57.633 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.633 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:34:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:34:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:34:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:34:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:34:57.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:34:57.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:34:57.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:34:57.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:34:57.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:34:57.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:34:57.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:34:57.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:34:57.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:34:57.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:34:57.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:34:57.638 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.639 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.639 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:57.639 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:57.639 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.847962686000002, 2024-04-24T14:34:57.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:34:57.639 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.639 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:57.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:57.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:57.639 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:34:57.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.640 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:57.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:34:57.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:34:57.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:57.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:34:57.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:34:57.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:34:57.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:34:57.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:34:57.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:34:57.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:34:57.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:34:57.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:57.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:34:57.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:34:57.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:34:57.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:34:57.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:57.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:34:57.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:34:57.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:34:57.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.645 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:57.645 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:57.645 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.847886103999997, 2024-04-24T14:34:57.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:34:57.645 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.645 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:57.645 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.645 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:57.645 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:57.645 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:34:57.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.645 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:57.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:34:57.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:34:57.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:57.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:34:57.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:34:57.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:34:57.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:34:57.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:34:57.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:34:57.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:34:57.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:34:57.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:57.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:34:57.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:34:57.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:34:57.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:34:57.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:57.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:34:57.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:34:57.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:34:57.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:34:57.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:34:57.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:34:57.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:34:57.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:34:57.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.650 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.651 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:57.651 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:57.651 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.847838558999996, 2024-04-24T14:34:57.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:34:57.651 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.651 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:57.651 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.651 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:57.651 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:57.651 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:34:57.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.651 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:57.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:34:57.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:34:57.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:34:57.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:34:57.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:34:57.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:34:57.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:57.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:34:57.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:34:57.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:34:57.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:34:57.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:57.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:34:57.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:34:57.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:34:57.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:34:57.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:34:57.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:34:57.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:34:57.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:34:57.656 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.656 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.656 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.656 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:57.657 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:57.657 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.847797916999994, 2024-04-24T14:34:57.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:34:57.657 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.657 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:57.657 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.657 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:57.657 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:57.657 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:34:57.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.657 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:57.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:57.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:34:57.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:57.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:34:57.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:34:57.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:34:57.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:34:57.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:34:57.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:34:57.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:34:57.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:34:57.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:57.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:34:57.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:34:57.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:34:57.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:34:57.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:57.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:34:57.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:34:57.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:34:57.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:34:57.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:34:57.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:34:57.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:34:57.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:34:57.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.662 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.662 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:57.662 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:57.663 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.847780806000003, 2024-04-24T14:34:57.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:34:57.663 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.663 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:57.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:57.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:57.663 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:34:57.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.663 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:57.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:34:57.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:34:57.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:57.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:34:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:34:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:34:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:34:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:34:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:34:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:34:57.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:34:57.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:57.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:34:57.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:34:57.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:34:57.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:34:57.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:34:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:34:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:34:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:34:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:34:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:34:57.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:34:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:34:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:103 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.648214693, 2024-04-24T14:34:57.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032646351999999997, 2024-04-24T14:34:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 103 2024-04-24T14:34:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959674+0000", 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959688+0000", 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959722+0000", 2024-04-24T14:34:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3936000000000002e-05 2024-04-24T14:34:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959735+0000", 2024-04-24T14:34:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2926e-05 2024-04-24T14:34:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959921+0000", 2024-04-24T14:34:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185601 2024-04-24T14:34:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960324+0000", 2024-04-24T14:34:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040356700000000002 2024-04-24T14:34:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960839+0000", 2024-04-24T14:34:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051410599999999996 2024-04-24T14:34:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:57.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.975634+0000", 2024-04-24T14:34:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014795770999999999 2024-04-24T14:34:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992113+0000", 2024-04-24T14:34:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016478982999999999 2024-04-24T14:34:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992228+0000", 2024-04-24T14:34:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114295 2024-04-24T14:34:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992323+0000", 2024-04-24T14:34:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5344000000000001e-05 2024-04-24T14:34:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:105 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.647964914999999, 2024-04-24T14:34:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032581757000000003, 2024-04-24T14:34:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.674 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105 2024-04-24T14:34:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959925+0000", 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959937+0000", 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2391999999999999e-05 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:34:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2535999999999999e-05 2024-04-24T14:34:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959970+0000", 2024-04-24T14:34:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1039999999999999e-05 2024-04-24T14:34:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969531+0000", 2024-04-24T14:34:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095603949999999993 2024-04-24T14:34:57.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969669+0000", 2024-04-24T14:34:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013806499999999999 2024-04-24T14:34:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970010+0000", 2024-04-24T14:34:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034062200000000001 2024-04-24T14:34:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976983+0000", 2024-04-24T14:34:57.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069735600000000002 2024-04-24T14:34:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992413+0000", 2024-04-24T14:34:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015429841999999999 2024-04-24T14:34:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992457+0000", 2024-04-24T14:34:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4169999999999999e-05 2024-04-24T14:34:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992508+0000", 2024-04-24T14:34:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0995000000000002e-05 2024-04-24T14:34:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:113 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.627738623999999, 2024-04-24T14:34:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014793863000000001, 2024-04-24T14:34:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:57.680 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 113 2024-04-24T14:34:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:34:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980152+0000", 2024-04-24T14:34:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980156+0000", 2024-04-24T14:34:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.619e-06 2024-04-24T14:34:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980163+0000", 2024-04-24T14:34:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7070000000000003e-06 2024-04-24T14:34:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980173+0000", 2024-04-24T14:34:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0957e-05 2024-04-24T14:34:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983381+0000", 2024-04-24T14:34:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032079930000000001 2024-04-24T14:34:57.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983628+0000", 2024-04-24T14:34:57.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024665700000000001 2024-04-24T14:34:57.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:57.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983964+0000", 2024-04-24T14:34:57.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000336332 2024-04-24T14:34:57.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:57.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992694+0000", 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087297009999999994 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994878+0000", 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021841439999999998 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994917+0000", 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8544999999999998e-05 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994947+0000", 2024-04-24T14:34:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9705000000000001e-05 2024-04-24T14:34:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:130 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.602045189000002, 2024-04-24T14:34:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014646289999999999, 2024-04-24T14:34:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:57.686 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 130 2024-04-24T14:34:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:34:57.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005844+0000", 2024-04-24T14:34:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005853+0000", 2024-04-24T14:34:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1319999999999994e-06 2024-04-24T14:34:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005870+0000", 2024-04-24T14:34:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7448999999999998e-05 2024-04-24T14:34:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005876+0000", 2024-04-24T14:34:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3319999999999997e-06 2024-04-24T14:34:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005913+0000", 2024-04-24T14:34:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6835999999999998e-05 2024-04-24T14:34:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006017+0000", 2024-04-24T14:34:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010358500000000001 2024-04-24T14:34:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006139+0000", 2024-04-24T14:34:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001227 2024-04-24T14:34:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.010194+0000", 2024-04-24T14:34:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040549380000000001 2024-04-24T14:34:57.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020365+0000", 2024-04-24T14:34:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010170983 2024-04-24T14:34:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020428+0000", 2024-04-24T14:34:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2457000000000002e-05 2024-04-24T14:34:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020492+0000", 2024-04-24T14:34:57.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4592999999999996e-05 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:155 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1601536~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.568384402, 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015144391, 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 155 2024-04-24T14:34:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:34:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039506+0000", 2024-04-24T14:34:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039766+0000", 2024-04-24T14:34:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025994400000000002 2024-04-24T14:34:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039780+0000", 2024-04-24T14:34:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3366e-05 2024-04-24T14:34:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039787+0000", 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4359999999999996e-06 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.050886+0000", 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011098993 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051026+0000", 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001395 2024-04-24T14:34:57.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051211+0000", 2024-04-24T14:34:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018547099999999999 2024-04-24T14:34:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054490+0000", 2024-04-24T14:34:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032785499999999999 2024-04-24T14:34:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054505+0000", 2024-04-24T14:34:57.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5906999999999999e-05 2024-04-24T14:34:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054594+0000", 2024-04-24T14:34:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8817e-05 2024-04-24T14:34:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054651+0000", 2024-04-24T14:34:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7040999999999998e-05 2024-04-24T14:34:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.697 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:156 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1732608~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.568093695999998, 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016895194999999998, 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 156 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.698 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039797+0000", 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:57.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040065+0000", 2024-04-24T14:34:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000268361 2024-04-24T14:34:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040086+0000", 2024-04-24T14:34:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1024000000000001e-05 2024-04-24T14:34:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040094+0000", 2024-04-24T14:34:57.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9379999999999992e-06 2024-04-24T14:34:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051936+0000", 2024-04-24T14:34:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011842518 2024-04-24T14:34:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052062+0000", 2024-04-24T14:34:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125482 2024-04-24T14:34:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052206+0000", 2024-04-24T14:34:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143953 2024-04-24T14:34:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054715+0000", 2024-04-24T14:34:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025088240000000002 2024-04-24T14:34:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056534+0000", 2024-04-24T14:34:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001819436 2024-04-24T14:34:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056610+0000", 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5383000000000007e-05 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056693+0000", 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3283999999999997e-05 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:157 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1863680~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:57.703 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.567786316999999, 2024-04-24T14:34:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016763674999999999, 2024-04-24T14:34:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157 2024-04-24T14:34:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:57.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040104+0000", 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040244+0000", 2024-04-24T14:34:57.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014035499999999999 2024-04-24T14:34:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040255+0000", 2024-04-24T14:34:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0135999999999999e-05 2024-04-24T14:34:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040260+0000", 2024-04-24T14:34:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7919999999999998e-06 2024-04-24T14:34:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053406+0000", 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013146023999999999 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053531+0000", 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012471800000000001 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053715+0000", 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018395099999999999 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:57.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056711+0000", 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029956520000000001 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056783+0000", 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2010999999999999e-05 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056831+0000", 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8189999999999998e-05 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056869+0000", 2024-04-24T14:34:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7812000000000003e-05 2024-04-24T14:34:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:158 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1994752~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.567624389999999, 2024-04-24T14:34:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016845334999999999, 2024-04-24T14:34:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.709 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2024-04-24T14:34:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:34:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040266+0000", 2024-04-24T14:34:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040379+0000", 2024-04-24T14:34:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112728 2024-04-24T14:34:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040387+0000", 2024-04-24T14:34:57.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8499999999999994e-06 2024-04-24T14:34:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040391+0000", 2024-04-24T14:34:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2719999999999999e-06 2024-04-24T14:34:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054803+0000", 2024-04-24T14:34:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014411548 2024-04-24T14:34:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054982+0000", 2024-04-24T14:34:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017941699999999999 2024-04-24T14:34:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:34:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.055167+0000", 2024-04-24T14:34:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018542100000000001 2024-04-24T14:34:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056718+0000", 2024-04-24T14:34:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015505670000000001 2024-04-24T14:34:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057020+0000", 2024-04-24T14:34:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030227500000000002 2024-04-24T14:34:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057070+0000", 2024-04-24T14:34:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9611000000000003e-05 2024-04-24T14:34:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057112+0000", 2024-04-24T14:34:57.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2373999999999998e-05 2024-04-24T14:34:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:159 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.562189402000001, 2024-04-24T14:34:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015930650000000001, 2024-04-24T14:34:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:34:57.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2024-04-24T14:34:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:34:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045701+0000", 2024-04-24T14:34:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045707+0000", 2024-04-24T14:34:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.562e-06 2024-04-24T14:34:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045728+0000", 2024-04-24T14:34:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1109999999999999e-05 2024-04-24T14:34:57.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045738+0000", 2024-04-24T14:34:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0016e-05 2024-04-24T14:34:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056183+0000", 2024-04-24T14:34:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010444836000000001 2024-04-24T14:34:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:57.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056470+0000", 2024-04-24T14:34:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028696899999999998 2024-04-24T14:34:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057160+0000", 2024-04-24T14:34:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069004699999999995 2024-04-24T14:34:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057291+0000", 2024-04-24T14:34:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013096200000000001 2024-04-24T14:34:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:57.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.061633+0000", 2024-04-24T14:34:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004341414 2024-04-24T14:34:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:57.720 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:57.720 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:57.721 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:57.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:34:57.723 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:57.723 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:57.723 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:57.724 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:57.724 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:57.724 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:57.724 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:57.724 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:57.724 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:57.724+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:57.822 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:57.828+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:57.873 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:57.874 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:57.874 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:57.874 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:57.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:34:57.923 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:57.926+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:57.982 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:57.983 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:57.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:34:58.024 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:58.026+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:58.090 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:58.090 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:34:58.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:34:58.090 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:34:58.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.090 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:58.090 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:58.090 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.622497707999997, 2024-04-24T14:34:58.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:34:58.091 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.091 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.091 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.091 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.091 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.091 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:34:58.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.091 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:58.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:34:58.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:34:58.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:58.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:34:58.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:34:58.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:34:58.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:34:58.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:34:58.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:34:58.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:34:58.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:34:58.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:34:58.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:34:58.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:34:58.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:34:58.095 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.095 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.095 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.095 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.095 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:58.095 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.247141005000003, 2024-04-24T14:34:58.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:34:58.095 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.095 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.096 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:34:58.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.096 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:58.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:34:58.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:34:58.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:58.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:34:58.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:34:58.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:34:58.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:34:58.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:34:58.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:58.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:34:58.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:34:58.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:34:58.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:34:58.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:34:58.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:34:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.245712850999993, 2024-04-24T14:34:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:34:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.100 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.101 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:34:58.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.101 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:58.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:34:58.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:34:58.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:58.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:34:58.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:34:58.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:34:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:34:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:34:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:34:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:34:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:34:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:34:58.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:34:58.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:34:58.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:34:58.104 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.104 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.104 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.104 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.105 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:58.105 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.245685761999994, 2024-04-24T14:34:58.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:34:58.105 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.105 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.105 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.105 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.105 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.105 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:34:58.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.105 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:58.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:34:58.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:34:58.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:58.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:34:58.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:34:58.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:34:58.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:34:58.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:34:58.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:34:58.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:34:58.109 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.109 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.109 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.109 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.109 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:58.109 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.245661795000004, 2024-04-24T14:34:58.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:34:58.109 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.109 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.109 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.109 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.109 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.110 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:34:58.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.110 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:58.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:34:58.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:34:58.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:58.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:34:58.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:34:58.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:34:58.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:34:58.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:34:58.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:34:58.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:34:58.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:34:58.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:34:58.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:34:58.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:34:58.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:34:58.113 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.113 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.113 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.113 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.113 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:58.113 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.231677816000001, 2024-04-24T14:34:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:34:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:34:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:34:58.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:34:58.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:58.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:34:58.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:34:58.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:34:58.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:34:58.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:34:58.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:34:58.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:34:58.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:34:58.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:34:58.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:34:58.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:34:58.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:34:58.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:34:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:34:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.231561505000002, 2024-04-24T14:34:58.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:34:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:34:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:34:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:34:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:34:58.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:34:58.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:34:58.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:34:58.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:34:58.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:34:58.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:34:58.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:34:58.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:34:58.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:34:58.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:34:58.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:34:58.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:34:58.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:34:58.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:34:58.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:34:58.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:34:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:34:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.231513847000002, 2024-04-24T14:34:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:34:58.125 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.126 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.126 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.126 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.126 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.126 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:34:58.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.126 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:58.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:34:58.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:34:58.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:58.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:34:58.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:34:58.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:34:58.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:34:58.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:34:58.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:34:58.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:34:58.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:34:58.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:34:58.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:34:58.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:34:58.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:34:58.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:34:58.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:34:58.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:34:58.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:34:58.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:34:58.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:34:58.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:34:58.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:34:58.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.131 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.131 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.131 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:58.131 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.227476549000002, 2024-04-24T14:34:58.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:34:58.131 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.131 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:34:58.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:34:58.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:58.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:34:58.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:34:58.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:34:58.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:34:58.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:34:58.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:34:58.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:58.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:34:58.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:34:58.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:34:58.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:34:58.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:34:58.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:34:58.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:34:58.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:34:58.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:34:58.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:34:58.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:34:58.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:34:58.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:34:58.137 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.137 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.137 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.137 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.137 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:58.137 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.227412790000002, 2024-04-24T14:34:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:34:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:34:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:58.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:34:58.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:34:58.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:58.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:34:58.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:34:58.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:34:58.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:34:58.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:34:58.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:34:58.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:34:58.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:34:58.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:34:58.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:34:58.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:34:58.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:34:58.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:34:58.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:34:58.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:34:58.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:34:58.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:34:58.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:34:58.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:34:58.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:34:58.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.143 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.144 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.144 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:58.144 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.227387300000004, 2024-04-24T14:34:58.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:34:58.144 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.144 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.144 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.144 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.144 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.144 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:34:58.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.144 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:58.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:58.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:34:58.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:58.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:34:58.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:34:58.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:34:58.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:34:58.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:34:58.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:34:58.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:34:58.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:34:58.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:34:58.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:34:58.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:34:58.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:34:58.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:34:58.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:34:58.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:34:58.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:34:58.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:34:58.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:34:58.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:34:58.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:34:58.149 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.149 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.149 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.150 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.150 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:58.150 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.227363550000007, 2024-04-24T14:34:58.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:34:58.150 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.150 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.150 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:34:58.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.150 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:58.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:34:58.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:34:58.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:58.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:34:58.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:34:58.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:34:58.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:34:58.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:34:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:34:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:34:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:34:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:34:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:34:58.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:34:58.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:34:58.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:34:58.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:34:58.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:34:58.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:34:58.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:34:58.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:34:58.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:34:58.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:34:58.155 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.227325037, 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:34:58.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.157 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:58.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:34:58.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:34:58.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:58.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:34:58.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:34:58.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:34:58.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:34:58.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:34:58.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:34:58.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:34:58.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:34:58.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:34:58.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:34:58.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:34:58.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:34:58.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:34:58.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:34:58.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:34:58.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:34:58.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:34:58.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:34:58.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:34:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:34:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.227296839999994, 2024-04-24T14:34:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:34:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:34:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:34:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:34:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:34:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:34:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:34:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:34:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:34:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:34:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:34:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:34:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:34:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:34:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:34:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:34:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:34:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:34:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:34:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:34:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:34:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:34:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:34:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:34:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.227275383000006, 2024-04-24T14:34:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:34:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:34:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:34:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:34:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:34:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:34:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:34:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:34:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:34:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:34:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:34:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:34:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:34:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:34:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:34:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:34:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:34:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:34:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:34:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:34:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:34:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:34:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:34:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:34:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:34:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.227256541000003, 2024-04-24T14:34:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:34:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:34:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:34:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:34:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:34:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:34:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:34:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:34:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:34:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:34:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:34:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:34:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:34:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:34:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:34:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:34:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:34:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.014424392000002, 2024-04-24T14:34:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:34:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:34:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:34:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:34:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:34:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:34:58.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:58.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:34:58.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:34:58.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:34:58.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:58.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:34:58.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:34:58.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:34:58.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:34:58.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:34:58.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:34:58.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:34:58.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:34:58.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:34:58.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:34:58.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:34:58.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:34:58.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:34:58.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:34:58.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:34:58.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:34:58.186 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.186 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.186 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.186 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:39 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1753088~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:58.186 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:58.186 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.074651975999998, 2024-04-24T14:34:58.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01139394, 2024-04-24T14:34:58.186 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.186 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.186 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.186 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:58.187 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:58.187 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 39 2024-04-24T14:34:58.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.187 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:58.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:34:58.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018967+0000", 2024-04-24T14:34:58.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:58.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019242+0000", 2024-04-24T14:34:58.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027504500000000002 2024-04-24T14:34:58.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019274+0000", 2024-04-24T14:34:58.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2472999999999999e-05 2024-04-24T14:34:58.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019288+0000", 2024-04-24T14:34:58.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3283e-05 2024-04-24T14:34:58.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026753+0000", 2024-04-24T14:34:58.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074651120000000003 2024-04-24T14:34:58.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026971+0000", 2024-04-24T14:34:58.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021810800000000001 2024-04-24T14:34:58.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:58.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.027211+0000", 2024-04-24T14:34:58.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023975 2024-04-24T14:34:58.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:58.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030200+0000", 2024-04-24T14:34:58.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029890709999999998 2024-04-24T14:34:58.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:58.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030252+0000", 2024-04-24T14:34:58.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1926999999999999e-05 2024-04-24T14:34:58.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:58.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030312+0000", 2024-04-24T14:34:58.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0602e-05 2024-04-24T14:34:58.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030363+0000", 2024-04-24T14:34:58.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0342e-05 2024-04-24T14:34:58.192 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.192 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.192 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.192 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:40 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1884160~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:58.192 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:58.192 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.074316002, 2024-04-24T14:34:58.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011199064, 2024-04-24T14:34:58.192 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.192 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 40 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019303+0000", 2024-04-24T14:34:58.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:58.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019729+0000", 2024-04-24T14:34:58.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042620899999999998 2024-04-24T14:34:58.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019769+0000", 2024-04-24T14:34:58.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9847e-05 2024-04-24T14:34:58.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019797+0000", 2024-04-24T14:34:58.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8347000000000002e-05 2024-04-24T14:34:58.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028299+0000", 2024-04-24T14:34:58.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085010629999999997 2024-04-24T14:34:58.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028385+0000", 2024-04-24T14:34:58.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6824999999999996e-05 2024-04-24T14:34:58.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:58.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028510+0000", 2024-04-24T14:34:58.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012483000000000001 2024-04-24T14:34:58.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:58.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030410+0000", 2024-04-24T14:34:58.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001899612 2024-04-24T14:34:58.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:58.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030416+0000", 2024-04-24T14:34:58.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7699999999999998e-06 2024-04-24T14:34:58.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:58.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030461+0000", 2024-04-24T14:34:58.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5157999999999997e-05 2024-04-24T14:34:58.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030504+0000", 2024-04-24T14:34:58.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2988999999999998e-05 2024-04-24T14:34:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:41 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 2015232~81920] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:34:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.073807295000002, 2024-04-24T14:34:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012345257, 2024-04-24T14:34:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:34:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:34:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:34:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:34:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 41 2024-04-24T14:34:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:34:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:34:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:34:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:34:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:34:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:34:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019812+0000", 2024-04-24T14:34:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:34:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019975+0000", 2024-04-24T14:34:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016325700000000001 2024-04-24T14:34:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:34:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020025+0000", 2024-04-24T14:34:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9484999999999998e-05 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020035+0000", 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0444e-05 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029150+0000", 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091150160000000001 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029352+0000", 2024-04-24T14:34:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020166499999999999 2024-04-24T14:34:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:34:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029611+0000", 2024-04-24T14:34:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025949600000000001 2024-04-24T14:34:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:34:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030518+0000", 2024-04-24T14:34:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090633300000000005 2024-04-24T14:34:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032088+0000", 2024-04-24T14:34:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015703290000000001 2024-04-24T14:34:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:34:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032130+0000", 2024-04-24T14:34:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2273e-05 2024-04-24T14:34:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:34:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:34:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:34:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032159+0000", 2024-04-24T14:34:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8387000000000001e-05 2024-04-24T14:34:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.204 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.204 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:34:58.204 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:34:58.204 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:58.205 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:34:58.206 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:58.206+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:58.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T14:34:58.306 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:58.308+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:58.314 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:58.314 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:58.314 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:58.314 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:58.324 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:34:58.408 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:58.410+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:58.419 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:58.419 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:58.419 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:58.420 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:58.420 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:58.429 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:34:58.508 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:58.512+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:58.527 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:58.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 155.904580767, 2024-04-24T14:34:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:34:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:34:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.528 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:34:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:34:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:34:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:34:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:34:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:34:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:34:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:34:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:34:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:34:58.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:34:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:34:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:34:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:34:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:34:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:34:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:34:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:34:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:34:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:34:58.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:34:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:34:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:58.534 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 155.90440368, 2024-04-24T14:34:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:34:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:34:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:58.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:34:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:34:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:58.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:34:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:34:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:34:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:34:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:34:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:34:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:34:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:34:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:34:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:34:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:34:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:34:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:34:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:34:58.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:34:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:34:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:34:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:34:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:34:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:34:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 155.90429976600001, 2024-04-24T14:34:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:34:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.541 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:34:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:34:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:34:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:34:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:34:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:34:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:34:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:34:58.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:34:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:34:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:34:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:34:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:34:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:34:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:34:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:34:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:34:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:58.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:34:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:34:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:34:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:34:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:34:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:34:58.547 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 155.90391131499999, 2024-04-24T14:34:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:34:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.548 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:34:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:34:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:34:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:34:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:34:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:34:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:34:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:34:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:34:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:34:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:34:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:34:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:34:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:34:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:34:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:34:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:34:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 155.903826103, 2024-04-24T14:34:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:34:58.554 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:34:58.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:34:58.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:58.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:34:58.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:34:58.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:34:58.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:34:58.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:34:58.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:34:58.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:34:58.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:34:58.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:34:58.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:34:58.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:58.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:34:58.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:34:58.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:58.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:34:58.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:34:58.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:58.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:34:58.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:34:58.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:58.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:34:58.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:34:58.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:34:58.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:34:58.560 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.560 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.560 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.560 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:58.560 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:58.560 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 155.90358922199999, 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:34:58.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:34:58.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:58.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.562 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:34:58.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:34:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:34:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:34:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:34:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:34:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:34:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:34:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:34:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:34:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:34:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:34:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:34:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:34:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:34:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:34:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:34:58.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:34:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:34:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:34:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:58.566 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 155.90350770000001, 2024-04-24T14:34:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:34:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:34:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:58.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:34:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:34:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:34:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:34:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:34:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:34:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:34:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:34:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:34:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:34:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:34:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:34:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:58.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:58.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:34:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:34:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:34:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:34:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:58.572 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 155.471595353, 2024-04-24T14:34:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:34:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:34:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:34:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:34:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:34:58.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:34:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:34:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:34:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:34:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:34:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:34:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:34:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:34:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:34:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:34:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:34:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:34:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:34:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:34:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:34:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:34:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:34:58.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:34:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:34:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 155.471567053, 2024-04-24T14:34:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:34:58.578 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:34:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:34:58.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:34:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:34:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:34:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:34:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:58.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:34:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:34:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:34:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:34:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:34:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:34:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 155.47147934899999, 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:58.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:34:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:34:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:34:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:34:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:34:58.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:34:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:34:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:34:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:34:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:34:58.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:34:58.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:34:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:34:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:34:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:34:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:34:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:34:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 155.47135654799999, 2024-04-24T14:34:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:34:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:34:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:34:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:34:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.590 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:34:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:34:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:34:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:34:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:34:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:34:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:34:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:34:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:34:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:34:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:34:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:34:58.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:34:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:34:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:34:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:34:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:34:58.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:34:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:34:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:34:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:34:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:34:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:34:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 86.165111988000007, 2024-04-24T14:34:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:34:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:34:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:34:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:34:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:34:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:34:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:58.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:34:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:34:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:34:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:34:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:34:58.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:34:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:34:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:34:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:34:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:34:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:34:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:34:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:34:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:34:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:34:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:34:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:34:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.600 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 78.470987988999994, 2024-04-24T14:34:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:34:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:34:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:34:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:34:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.601 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:58.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:34:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:34:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:34:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:34:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:34:58.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:34:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:34:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:34:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:34:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:34:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.511628023, 2024-04-24T14:34:58.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:34:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:34:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:34:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:34:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:34:58.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:34:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:34:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:34:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:34:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:34:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:34:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:34:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:34:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:34:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:34:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:34:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:34:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:34:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.511547468000003, 2024-04-24T14:34:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:34:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:34:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:34:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:34:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:34:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:34:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:34:58.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:34:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:34:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:34:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:34:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:34:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:34:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:34:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:34:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:34:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:34:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:34:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:34:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:34:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:34:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:34:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.619 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.511499739000001, 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:58.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:34:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:34:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:34:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:34:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:34:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:34:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:34:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:34:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:34:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:34:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:34:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:34:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:34:58.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:34:58.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:34:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:34:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:34:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.511471976000003, 2024-04-24T14:34:58.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:34:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:34:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:34:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:34:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:34:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:34:58.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:34:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:34:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:34:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:34:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:34:58.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:34:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:34:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:34:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:34:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:34:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:34:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:34:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:34:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:34:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:34:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:34:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:34:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:34:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.511432540999998, 2024-04-24T14:34:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:34:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.631 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:34:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:34:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:34:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:34:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:34:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:34:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:34:58.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:34:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:34:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:34:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:34:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:34:58.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:34:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:34:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:34:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:34:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:34:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.636 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:34:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:34:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.511407556999998, 2024-04-24T14:34:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:34:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.637 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:34:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:34:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:34:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:34:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:34:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:34:58.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:34:58.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:34:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:34:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:34:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:34:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:34:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:34:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:34:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:34:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:34:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:34:58.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:34:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:34:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 59.511381898000003, 2024-04-24T14:34:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:34:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:34:58.643 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:34:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:34:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:34:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:58.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:34:58.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:34:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:34:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:34:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:34:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:34:58.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:34:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:34:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:34:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:34:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:34:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:34:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:34:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.648 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.649 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:58.649 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:58.649 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:58.650 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:34:58.651 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:58.656+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:58.751 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:58.756+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:58.798 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:58.798 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:58.798 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:58.798 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:58.808 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:34:58.852 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:58.854+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:58.912 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:58.913 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:58.913 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:58.913 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:58.913 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:58.922 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:34:58.952 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:58.954+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:59.023 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:59.023 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:59.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:59.023 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:59.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.023 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:59.023 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:59.023 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.362377752, 2024-04-24T14:34:59.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:34:59.024 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.024 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:59.024 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.024 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:59.024 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:59.024 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:34:59.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.024 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.024 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.024 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:59.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:34:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:34:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:34:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:34:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:34:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:34:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:34:59.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:34:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:34:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:34:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:34:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:34:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:34:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.359343225999993, 2024-04-24T14:34:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:34:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:59.028 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:34:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:34:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:34:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:34:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:34:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:34:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:34:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:34:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:34:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:34:59.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:34:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:34:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:34:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:34:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.359319040000003, 2024-04-24T14:34:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:34:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:34:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.033 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:34:59.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:34:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:34:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:34:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:34:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:34:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:34:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:34:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:34:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:34:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:34:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:34:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:34:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.359301656, 2024-04-24T14:34:59.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:34:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:34:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:34:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:34:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.039 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:34:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:34:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:34:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:34:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:34:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:34:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.040 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:34:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:34:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:34:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:34:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:34:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:34:59.041 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.344865564000003, 2024-04-24T14:34:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:34:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:59.042 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.043 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:34:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:34:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:34:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:34:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:34:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:34:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:34:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:34:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:34:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.045 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:34:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:34:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:34:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:34:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.046 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.344833265000005, 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:59.047 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:34:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:34:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.048 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:34:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:34:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:34:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:34:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:34:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:34:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.049 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:34:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:34:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:34:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:34:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:34:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:34:59.050 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.330580987999994, 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:34:59.051 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:34:59.052 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:34:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:34:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:34:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:34:59.053 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:34:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:34:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:34:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:34:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.054 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:34:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:34:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:34:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:34:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.055 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.330551888000002, 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:59.056 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:34:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.057 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:34:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:34:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:34:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:34:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:34:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:34:59.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:34:59.059 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.330525957999996, 2024-04-24T14:34:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:34:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:59.060 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:34:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:34:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:34:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:34:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:34:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:34:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:34:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:34:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:34:59.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:34:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:34:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:34:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:34:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:34:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.064 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.314805493999998, 2024-04-24T14:34:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:34:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:34:59.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:34:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:34:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:34:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:34:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:34:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:34:59.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:34:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:34:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:34:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:34:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:34:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:34:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:34:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:34:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.069 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.314773561999999, 2024-04-24T14:34:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:34:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:34:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:34:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:34:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:59.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:34:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:34:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:34:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:34:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:34:59.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:34:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:34:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:34:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:34:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:34:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.073 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:34:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.314706045999998, 2024-04-24T14:34:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:34:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:34:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:34:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:34:59.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:34:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:34:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:34:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:34:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:34:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:34:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:34:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:34:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:34:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:34:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:34:59.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:34:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:34:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.257293163, 2024-04-24T14:34:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:34:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.078 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:59.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:34:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:34:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:34:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:34:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:34:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:34:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:34:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:34:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:103 3.1 e87/74 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head v 87'261, mlcod=87'258)", 2024-04-24T14:34:59.082 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.066011959000001, 2024-04-24T14:34:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019745728000000001, 2024-04-24T14:34:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.083 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960679+0000", 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960692+0000", 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3709999999999999e-05 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960722+0000", 2024-04-24T14:34:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9791999999999999e-05 2024-04-24T14:34:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960733+0000", 2024-04-24T14:34:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.128e-05 2024-04-24T14:34:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960857+0000", 2024-04-24T14:34:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000123981 2024-04-24T14:34:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960926+0000", 2024-04-24T14:34:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8881999999999995e-05 2024-04-24T14:34:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980353+0000", 2024-04-24T14:34:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019426853000000001 2024-04-24T14:34:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980426+0000", 2024-04-24T14:34:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2803000000000001e-05 2024-04-24T14:34:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:105 3.1 e87/74 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head v 87'263, mlcod=87'258)", 2024-04-24T14:34:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.056885424999997, 2024-04-24T14:34:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014864743999999999, 2024-04-24T14:34:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.087 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:34:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969805+0000", 2024-04-24T14:34:59.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969820+0000", 2024-04-24T14:34:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5116999999999999e-05 2024-04-24T14:34:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969852+0000", 2024-04-24T14:34:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2135999999999999e-05 2024-04-24T14:34:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969864+0000", 2024-04-24T14:34:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.189e-05 2024-04-24T14:34:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972122+0000", 2024-04-24T14:34:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022580389999999999 2024-04-24T14:34:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972165+0000", 2024-04-24T14:34:59.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2429000000000003e-05 2024-04-24T14:34:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984619+0000", 2024-04-24T14:34:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012454735 2024-04-24T14:34:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984672+0000", 2024-04-24T14:34:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2077999999999998e-05 2024-04-24T14:34:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:112 3.1 e87/74 3:a983798d:::rbd_data.12ff64619e2d.0000000000000006:head v 87'265, mlcod=87'258)", 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.046546589000002, 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012821978, 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:34:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980144+0000", 2024-04-24T14:34:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980159+0000", 2024-04-24T14:34:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4769999999999999e-05 2024-04-24T14:34:59.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980189+0000", 2024-04-24T14:34:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0419000000000001e-05 2024-04-24T14:34:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980213+0000", 2024-04-24T14:34:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4036000000000002e-05 2024-04-24T14:34:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980452+0000", 2024-04-24T14:34:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00023921500000000001 2024-04-24T14:34:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980468+0000", 2024-04-24T14:34:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5843999999999999e-05 2024-04-24T14:34:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992932+0000", 2024-04-24T14:34:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012463314 2024-04-24T14:34:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992968+0000", 2024-04-24T14:34:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6075999999999998e-05 2024-04-24T14:34:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:113 3.1 e87/74 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head v 87'267, mlcod=87'258)", 2024-04-24T14:34:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.043011864999997, 2024-04-24T14:34:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010561022999999999, 2024-04-24T14:34:59.096 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:34:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983679+0000", 2024-04-24T14:34:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983699+0000", 2024-04-24T14:34:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0392000000000001e-05 2024-04-24T14:34:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983737+0000", 2024-04-24T14:34:59.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8340000000000002e-05 2024-04-24T14:34:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983756+0000", 2024-04-24T14:34:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8402000000000001e-05 2024-04-24T14:34:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984706+0000", 2024-04-24T14:34:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095024400000000005 2024-04-24T14:34:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984741+0000", 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4721000000000002e-05 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994211+0000", 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094708559999999997 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994241+0000", 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9941e-05 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:115 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.042557828, 2024-04-24T14:34:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010347242, 2024-04-24T14:34:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:34:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:34:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.101 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:34:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984132+0000", 2024-04-24T14:34:59.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984142+0000", 2024-04-24T14:34:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7370000000000001e-06 2024-04-24T14:34:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984154+0000", 2024-04-24T14:34:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2288e-05 2024-04-24T14:34:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984163+0000", 2024-04-24T14:34:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3749999999999992e-06 2024-04-24T14:34:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984246+0000", 2024-04-24T14:34:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2962999999999998e-05 2024-04-24T14:34:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984473+0000", 2024-04-24T14:34:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00022696399999999999 2024-04-24T14:34:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984771+0000", 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00029739499999999999 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992998+0000", 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082275750000000009 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994299+0000", 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013005289999999999 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994403+0000", 2024-04-24T14:34:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010376499999999999 2024-04-24T14:34:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994482+0000", 2024-04-24T14:34:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9013999999999999e-05 2024-04-24T14:34:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.106 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:128 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:34:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.021366899999997, 2024-04-24T14:34:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011784144, 2024-04-24T14:34:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:34:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:34:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 128 2024-04-24T14:34:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005324+0000", 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005332+0000", 2024-04-24T14:34:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2989999999999997e-06 2024-04-24T14:34:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005354+0000", 2024-04-24T14:34:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1705999999999999e-05 2024-04-24T14:34:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005362+0000", 2024-04-24T14:34:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5599999999999996e-06 2024-04-24T14:34:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005418+0000", 2024-04-24T14:34:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6685999999999999e-05 2024-04-24T14:34:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005535+0000", 2024-04-24T14:34:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117008 2024-04-24T14:34:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:34:59.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005732+0000", 2024-04-24T14:34:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019718700000000001 2024-04-24T14:34:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:34:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016440+0000", 2024-04-24T14:34:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010707787999999999 2024-04-24T14:34:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:34:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016913+0000", 2024-04-24T14:34:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047270400000000001 2024-04-24T14:34:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017047+0000", 2024-04-24T14:34:59.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013434699999999999 2024-04-24T14:34:59.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017109+0000", 2024-04-24T14:34:59.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2192000000000004e-05 2024-04-24T14:34:59.112 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.112 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.112 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.113 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:130 3.1 e87/74 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 87'269, mlcod=87'265)", 2024-04-24T14:34:59.113 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:59.113 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 32.020914140000002, 2024-04-24T14:34:59.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013945631999999999, 2024-04-24T14:34:59.113 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.113 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.113 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.113 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:59.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:34:59.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005777+0000", 2024-04-24T14:34:59.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:59.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005791+0000", 2024-04-24T14:34:59.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3743999999999999e-05 2024-04-24T14:34:59.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005803+0000", 2024-04-24T14:34:59.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2330999999999999e-05 2024-04-24T14:34:59.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005810+0000", 2024-04-24T14:34:59.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8990000000000004e-06 2024-04-24T14:34:59.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005888+0000", 2024-04-24T14:34:59.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7872000000000003e-05 2024-04-24T14:34:59.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005911+0000", 2024-04-24T14:34:59.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3156e-05 2024-04-24T14:34:59.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019642+0000", 2024-04-24T14:34:59.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013730456 2024-04-24T14:34:59.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019724+0000", 2024-04-24T14:34:59.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1981e-05 2024-04-24T14:34:59.117 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.117 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.117 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.117 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.117 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.117 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:59.119 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:34:59.119 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:34:59.122+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:34:59.220 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:59.224+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:59.222 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:59.223 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:59.223 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:34:59.223 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:59.232 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:34:59.278 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-24T14:34:59.321 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:59.324+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:59.326 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:59.326 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:34:59.326 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:34:59.326 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:34:59.326 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:59.336 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:34:59.422 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:34:59.426+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:34:59.490 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:34:59.490 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:34:59.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:34:59.491 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:34:59.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.491 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:34:59.491 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:59.491 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 79.422327030999995, 2024-04-24T14:34:59.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:34:59.491 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.491 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.491 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:59.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:34:59.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:34:59.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:34:59.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:34:59.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:34:59.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:34:59.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:34:59.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:34:59.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:34:59.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:34:59.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:34:59.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:34:59.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:34:59.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:34:59.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:34:59.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:34:59.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:34:59.496 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.497 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.497 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.497 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:34:59.497 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:59.497 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.427512452000002, 2024-04-24T14:34:59.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:34:59.497 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.497 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.498 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:59.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:34:59.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:34:59.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:59.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:34:59.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:34:59.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:34:59.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:34:59.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:34:59.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:34:59.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:34:59.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:34:59.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:34:59.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:34:59.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:34:59.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:34:59.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:34:59.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:34:59.503 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.503 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.503 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.503 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:34:59.503 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:59.503 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.416583193999998, 2024-04-24T14:34:59.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:34:59.504 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.504 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.504 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:59.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:34:59.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:34:59.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:59.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:34:59.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:34:59.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:34:59.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:34:59.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:34:59.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:34:59.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:34:59.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:34:59.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:34:59.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:34:59.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:34:59.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:34:59.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:34:59.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:34:59.509 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.509 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.509 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.509 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:34:59.509 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:59.509 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.405627112999994, 2024-04-24T14:34:59.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:34:59.509 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.510 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.510 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:59.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:34:59.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:34:59.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:59.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:34:59.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:34:59.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:34:59.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:34:59.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:34:59.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:34:59.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:34:59.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:34:59.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:34:59.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:34:59.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:34:59.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:34:59.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:34:59.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:34:59.515 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.515 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.515 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.515 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:34:59.515 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:59.515 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.394057747999994, 2024-04-24T14:34:59.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:34:59.516 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.516 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.516 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:59.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:34:59.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:34:59.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:59.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:34:59.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:34:59.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:34:59.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:34:59.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:34:59.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:34:59.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:34:59.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:34:59.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:34:59.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:34:59.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:34:59.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:34:59.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:34:59.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:34:59.521 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.521 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.521 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.521 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:34:59.521 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:59.521 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.140938089000002, 2024-04-24T14:34:59.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:34:59.522 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.522 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.522 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:59.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:34:59.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:34:59.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:59.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:34:59.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:34:59.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:34:59.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:34:59.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:34:59.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:34:59.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:34:59.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:34:59.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:34:59.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:34:59.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:34:59.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:34:59.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:34:59.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:34:59.527 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.527 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.527 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.527 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:34:59.527 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:59.527 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.130192921000003, 2024-04-24T14:34:59.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:34:59.527 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.528 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.528 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:59.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:34:59.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:34:59.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:59.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:34:59.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:34:59.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:34:59.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:34:59.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:34:59.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:34:59.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:34:59.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:34:59.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:34:59.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:34:59.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:34:59.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:34:59.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:34:59.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:34:59.532 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.532 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.533 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.533 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:34:59.533 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:59.533 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.117826644999994, 2024-04-24T14:34:59.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:34:59.533 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.533 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.533 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:59.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:34:59.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:34:59.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:59.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:34:59.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:34:59.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:34:59.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:34:59.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:34:59.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:34:59.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:34:59.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:34:59.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:34:59.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:34:59.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:34:59.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:34:59.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:34:59.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:34:59.538 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.538 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.538 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.539 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:34:59.539 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:59.539 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.992890524000003, 2024-04-24T14:34:59.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:34:59.539 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.539 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.539 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:59.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:34:59.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:34:59.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:59.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:34:59.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:34:59.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:34:59.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:34:59.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:34:59.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:34:59.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:34:59.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:34:59.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:34:59.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:34:59.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:34:59.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:34:59.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:34:59.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:34:59.544 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.544 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.544 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.544 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:34:59.544 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:59.545 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.973502680999999, 2024-04-24T14:34:59.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:34:59.545 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.545 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.545 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:59.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:34:59.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:34:59.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:59.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:34:59.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:34:59.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.547 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.547 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:34:59.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:34:59.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:34:59.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:34:59.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:34:59.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:34:59.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:34:59.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:34:59.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:34:59.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:34:59.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:34:59.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:34:59.550 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.550 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.550 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.550 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:34:59.551 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:59.551 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.963774838000006, 2024-04-24T14:34:59.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:34:59.551 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.551 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.551 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:59.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:34:59.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:34:59.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:59.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:34:59.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:34:59.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:34:59.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:34:59.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:34:59.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:34:59.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:34:59.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:34:59.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:34:59.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:34:59.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:34:59.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:34:59.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:34:59.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:34:59.556 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.556 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.556 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.556 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:34:59.556 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:59.557 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.803665158000001, 2024-04-24T14:34:59.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:34:59.557 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.557 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.557 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:59.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:34:59.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:34:59.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:59.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:34:59.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:34:59.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:34:59.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:34:59.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:34:59.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:34:59.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:34:59.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:34:59.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:34:59.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:34:59.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:34:59.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:34:59.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.562 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:34:59.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:34:59.562 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.562 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.562 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.562 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:34:59.562 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:59.562 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.793936161999994, 2024-04-24T14:34:59.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:34:59.563 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.563 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.563 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:59.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:34:59.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:34:59.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:34:59.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:34:59.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:34:59.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:34:59.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:34:59.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:34:59.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:34:59.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:34:59.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:34:59.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:34:59.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:34:59.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:34:59.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:34:59.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:34:59.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:34:59.568 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.568 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.568 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.568 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:34:59.568 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:59.568 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 60.458965327000001, 2024-04-24T14:34:59.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:34:59.568 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.568 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:34:59.568 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:59.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:59.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:34:59.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:59.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:34:59.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:34:59.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:34:59.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:34:59.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:34:59.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:34:59.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:34:59.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:34:59.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:34:59.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:34:59.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:34:59.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:34:59.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:34:59.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:34:59.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:34:59.573 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.573 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.573 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.574 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:34:59.574 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:59.574 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 56.614814578999997, 2024-04-24T14:34:59.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:34:59.574 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.574 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:34:59.574 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:59.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:59.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:34:59.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:34:59.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:34:59.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:34:59.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:34:59.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:34:59.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:34:59.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:34:59.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:34:59.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:34:59.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:34:59.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:34:59.578 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.578 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.578 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.578 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:5 3.2 3.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:34:59.578 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:59.579 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.286576937, 2024-04-24T14:34:59.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043717790000000001, 2024-04-24T14:34:59.579 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.579 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-24T14:34:59.579 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:34:59.579 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:34:59.579 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:34:59.579 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 5 2024-04-24T14:34:59.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.579 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:59.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:34:59.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207678+0000", 2024-04-24T14:34:59.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999962 2024-04-24T14:34:59.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207700+0000", 2024-04-24T14:34:59.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1313000000000001e-05 2024-04-24T14:34:59.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207757+0000", 2024-04-24T14:34:59.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7499e-05 2024-04-24T14:34:59.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207779+0000", 2024-04-24T14:34:59.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1478999999999999e-05 2024-04-24T14:34:59.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.212054+0000", 2024-04-24T14:34:59.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042750849999999996 2024-04-24T14:34:59.583 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.583 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.583 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.583 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:34:59.583 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:59.583 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.391051578999999, 2024-04-24T14:34:59.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044084859999999997, 2024-04-24T14:34:59.584 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.584 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:59.584 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:59.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:34:59.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103206+0000", 2024-04-24T14:34:59.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:59.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104731+0000", 2024-04-24T14:34:59.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001524755 2024-04-24T14:34:59.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104759+0000", 2024-04-24T14:34:59.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7973000000000001e-05 2024-04-24T14:34:59.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104807+0000", 2024-04-24T14:34:59.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8112000000000003e-05 2024-04-24T14:34:59.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104907+0000", 2024-04-24T14:34:59.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100317 2024-04-24T14:34:59.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104918+0000", 2024-04-24T14:34:59.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747999999999999e-05 2024-04-24T14:34:59.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.107616+0000", 2024-04-24T14:34:59.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026977559999999999 2024-04-24T14:34:59.588 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.588 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.588 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.589 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(3.2 83/82 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:59.589 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:59.589 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.384657202, 2024-04-24T14:34:59.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041425760000000002, 2024-04-24T14:34:59.589 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.589 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:59.589 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.589 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:59.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:34:59.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109600+0000", 2024-04-24T14:34:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111210+0000", 2024-04-24T14:34:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016097279999999999 2024-04-24T14:34:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111237+0000", 2024-04-24T14:34:59.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7189999999999999e-05 2024-04-24T14:34:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111251+0000", 2024-04-24T14:34:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.365e-05 2024-04-24T14:34:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111367+0000", 2024-04-24T14:34:59.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000115703 2024-04-24T14:34:59.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111378+0000", 2024-04-24T14:34:59.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1873e-05 2024-04-24T14:34:59.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.113744+0000", 2024-04-24T14:34:59.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023657069999999999 2024-04-24T14:34:59.594 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.594 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.594 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.594 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:59.594 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:59.594 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.367756626000002, 2024-04-24T14:34:59.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046193859999999996, 2024-04-24T14:34:59.594 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.595 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:59.595 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.595 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.595 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:59.595 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.595 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:34:59.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.596 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126501+0000", 2024-04-24T14:34:59.596 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:34:59.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.596 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.596 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128169+0000", 2024-04-24T14:34:59.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016686229999999999 2024-04-24T14:34:59.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128202+0000", 2024-04-24T14:34:59.597 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2539000000000001e-05 2024-04-24T14:34:59.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.597 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.597 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128222+0000", 2024-04-24T14:34:59.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0149e-05 2024-04-24T14:34:59.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128375+0000", 2024-04-24T14:34:59.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000153459 2024-04-24T14:34:59.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128399+0000", 2024-04-24T14:34:59.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3393000000000001e-05 2024-04-24T14:34:59.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.131122+0000", 2024-04-24T14:34:59.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027227140000000002 2024-04-24T14:34:59.599 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.599 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.599 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.600 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 49'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@49'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:34:59.600 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:59.600 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.345975246000002, 2024-04-24T14:34:59.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038414529999999999, 2024-04-24T14:34:59.600 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:34:59.600 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:34:59.600 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:34:59.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:34:59.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:59.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:34:59.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:34:59.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:34:59.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:34:59.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148282+0000", 2024-04-24T14:34:59.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:34:59.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:34:59.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149875+0000", 2024-04-24T14:34:59.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015929900000000001 2024-04-24T14:34:59.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:34:59.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149908+0000", 2024-04-24T14:34:59.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3102000000000003e-05 2024-04-24T14:34:59.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:34:59.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149941+0000", 2024-04-24T14:34:59.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2480000000000001e-05 2024-04-24T14:34:59.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:34:59.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150031+0000", 2024-04-24T14:34:59.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0394000000000002e-05 2024-04-24T14:34:59.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:34:59.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150043+0000", 2024-04-24T14:34:59.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1967000000000001e-05 2024-04-24T14:34:59.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:34:59.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:34:59.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:34:59.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.152125+0000", 2024-04-24T14:34:59.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020817029999999999 2024-04-24T14:34:59.604 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.604 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.604 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.605 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:34:59.605 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:34:59.605 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:34:59.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:34:59.607 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:34:59.610+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:34:59.707 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:34:59.712+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:34:59.718 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:59.719 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:59.719 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:34:59.719 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:59.725 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:34:59.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:34:59.807 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:34:59.812+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:34:59.862 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:34:59.876 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:34:59.876 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:34:59.876 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:34:59.876 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:34:59.876 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:34:59.880 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:34:59.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:34:59.899 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:34:59.908 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:34:59.912+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:34:59.919 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:34:59.942 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:34:59.964 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:34:59.967 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:34:59.967 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:34:59.967 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:34:59.967 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:34:59.967 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:34:59.967 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:34:59.983 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:34:59.983 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:34:59.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:35:00.009 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:00.012+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:00.109 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:00.110+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:00.134 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:00.134 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:00.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:35:00.210 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:00.214+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:00.239 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:00.239 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:00.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:35:00.311 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:00.314+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:00.389 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:00.389 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:00.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:35:00.412 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:00.416+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:00.493 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:00.493 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:00.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:35:00.513 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:00.516+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:00.598 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:00.598 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:00.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:35:00.613 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:00.614+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:00.704 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:00.704 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:00.714 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:35:00.714 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:00.718+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:00.815 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:00.820+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:00.822 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:00.822 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:00.832 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:35:00.915 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:00.920+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:00.926 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:00.926 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:00.935 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:35:01.016 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:01.020+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:01.031 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:01.031 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:01.041 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:35:01.118 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:01.120+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:01.139 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:01.139 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:01.148 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:35:01.219 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:01.224+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:01.247 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:01.247 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:01.256 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:35:01.291 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T14:35:01.338 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:01.342+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:01.402 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:01.402 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:01.411 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:35:01.420 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:01.422+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:01.509 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:01.509 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:01.518 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:35:01.521 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:01.524+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:01.614 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:01.614 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:01.622 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:01.624+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:01.623 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:35:01.723 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:01.726+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:01.725 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:01.725 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:01.734 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:35:01.824 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:01.826+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:01.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:35:01.925 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:01.926+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:01.950 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:01.950 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:01.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:35:01.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:35:02.026 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:02.032+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:02.118 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:35:02.127 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:02.130+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:35:02.162 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:35:02.181 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:35:02.201 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:35:02.212 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:02.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:35:02.223 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:35:02.245 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:35:02.246 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:02.250+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:02.248 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:35:02.248 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:35:02.248 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:35:02.249 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:35:02.249 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:35:02.249 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:35:02.288 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-24T14:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:02.317 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:02.320 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4] out_osds: [5, 1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 1] 2024-04-24T14:35:02.320 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:35:02.320 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-04-24T14:35:02.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-04-24T14:35:02.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:35:02.370 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:02.374+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:02.430 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:02.434+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:02.476 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:02.476 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:02.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:02.476 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:02.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.476 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.476 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:02.476 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 80.722831174000007, 2024-04-24T14:35:02.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:35:02.476 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.476 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.477 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:35:02.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.477 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:02.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:02.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:35:02.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:02.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:35:02.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:35:02.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:35:02.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:35:02.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:35:02.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:35:02.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:35:02.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:35:02.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:02.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:35:02.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:35:02.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:35:02.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:35:02.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:02.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:35:02.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:35:02.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:35:02.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:35:02.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:35:02.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:35:02.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:35:02.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:35:02.482 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.482 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.482 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.482 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.482 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:02.483 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 80.72278034, 2024-04-24T14:35:02.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:35:02.483 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.483 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.483 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.483 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.483 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.483 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:35:02.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.483 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:35:02.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:35:02.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:35:02.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:35:02.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:35:02.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:35:02.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:35:02.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:35:02.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:02.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:35:02.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:35:02.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:35:02.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:35:02.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:02.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:35:02.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:35:02.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:35:02.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:35:02.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:35:02.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:35:02.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:35:02.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:35:02.488 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.488 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.488 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.488 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.488 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:02.488 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 80.722744032999998, 2024-04-24T14:35:02.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:35:02.489 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.489 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.489 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:35:02.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.489 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:02.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:02.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:35:02.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:02.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:35:02.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:35:02.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:35:02.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:35:02.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:35:02.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:35:02.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:35:02.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:35:02.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:02.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:35:02.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:35:02.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:35:02.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:35:02.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:02.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:35:02.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:35:02.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:35:02.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:35:02.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 80.719332782999999, 2024-04-24T14:35:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:35:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:35:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.495 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:35:02.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:35:02.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:35:02.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:35:02.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:02.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:35:02.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:35:02.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:35:02.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 80.719271105999994, 2024-04-24T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:35:02.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:35:02.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:02.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:35:02.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:35:02.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:35:02.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 80.719236530000003, 2024-04-24T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:35:02.507 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.508 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.508 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:35:02.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.508 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:02.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:02.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:02.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:02.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:35:02.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:35:02.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:35:02.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:35:02.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:35:02.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:35:02.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:35:02.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:35:02.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:02.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:35:02.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:35:02.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:35:02.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:35:02.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:02.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:35:02.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:35:02.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:35:02.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:35:02.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:35:02.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:35:02.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:35:02.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:35:02.513 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.513 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.513 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.513 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.514 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:02.514 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 80.719207190999995, 2024-04-24T14:35:02.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:35:02.514 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.514 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.514 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:35:02.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.514 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:02.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:02.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:35:02.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:02.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:35:02.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:35:02.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:35:02.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:35:02.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:35:02.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:35:02.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:35:02.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:02.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:35:02.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:35:02.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:35:02.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:35:02.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:35:02.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:35:02.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:35:02.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:35:02.519 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.519 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.519 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 80.719130609000004, 2024-04-24T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:02.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:35:02.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:35:02.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:35:02.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:35:02.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:35:02.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:35:02.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:35:02.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:35:02.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:02.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:35:02.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:35:02.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:35:02.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:35:02.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:02.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:35:02.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:35:02.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:35:02.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:35:02.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:35:02.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:35:02.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:35:02.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:35:02.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.526 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.526 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.526 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:02.526 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 80.719083064000003, 2024-04-24T14:35:02.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:35:02.526 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.526 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.526 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.526 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.527 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.527 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:35:02.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.527 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:02.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:02.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:35:02.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:02.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:35:02.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:35:02.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:35:02.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:35:02.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:35:02.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:35:02.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:35:02.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:35:02.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:02.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:35:02.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:35:02.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:35:02.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:35:02.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:02.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:35:02.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:35:02.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:35:02.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:35:02.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:35:02.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:35:02.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:35:02.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:35:02.532 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.532 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.532 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.532 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.532 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:02.532 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 80.719042422000001, 2024-04-24T14:35:02.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:35:02.532 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:35:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:35:02.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:35:02.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:35:02.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:35:02.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:35:02.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:35:02.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:35:02.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:35:02.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:02.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:35:02.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:35:02.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:35:02.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:35:02.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:02.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:35:02.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:35:02.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:35:02.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:35:02.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:35:02.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:35:02.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:35:02.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:35:02.538 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.538 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.538 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.538 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.539 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:02.539 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 80.719025310999996, 2024-04-24T14:35:02.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:35:02.539 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.539 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.539 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.539 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.539 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.539 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:35:02.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.539 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:02.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:02.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:35:02.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:02.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:35:02.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:35:02.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:35:02.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:02.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:35:02.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:35:02.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:35:02.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:35:02.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:02.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:35:02.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:35:02.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:35:02.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:35:02.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:35:02.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:35:02.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:35:02.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:35:02.544 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.544 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.544 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.544 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:103 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:02.545 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:02.545 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.519459198, 2024-04-24T14:35:02.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032646351999999997, 2024-04-24T14:35:02.545 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.545 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:02.545 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.545 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:02.545 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:02.545 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 103 2024-04-24T14:35:02.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.545 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:02.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:02.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959674+0000", 2024-04-24T14:35:02.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:02.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959688+0000", 2024-04-24T14:35:02.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2024-04-24T14:35:02.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959722+0000", 2024-04-24T14:35:02.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3936000000000002e-05 2024-04-24T14:35:02.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959735+0000", 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2926e-05 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959921+0000", 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185601 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960324+0000", 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040356700000000002 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960839+0000", 2024-04-24T14:35:02.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051410599999999996 2024-04-24T14:35:02.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:02.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.975634+0000", 2024-04-24T14:35:02.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014795770999999999 2024-04-24T14:35:02.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:02.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992113+0000", 2024-04-24T14:35:02.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016478982999999999 2024-04-24T14:35:02.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:02.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992228+0000", 2024-04-24T14:35:02.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114295 2024-04-24T14:35:02.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992323+0000", 2024-04-24T14:35:02.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5344000000000001e-05 2024-04-24T14:35:02.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.550 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.550 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:105 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:02.550 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:02.551 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.519209420000003, 2024-04-24T14:35:02.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032581757000000003, 2024-04-24T14:35:02.551 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.551 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:02.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:02.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:02.551 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105 2024-04-24T14:35:02.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.551 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959925+0000", 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959937+0000", 2024-04-24T14:35:02.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2391999999999999e-05 2024-04-24T14:35:02.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:35:02.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2535999999999999e-05 2024-04-24T14:35:02.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959970+0000", 2024-04-24T14:35:02.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1039999999999999e-05 2024-04-24T14:35:02.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969531+0000", 2024-04-24T14:35:02.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095603949999999993 2024-04-24T14:35:02.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969669+0000", 2024-04-24T14:35:02.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013806499999999999 2024-04-24T14:35:02.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:02.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970010+0000", 2024-04-24T14:35:02.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034062200000000001 2024-04-24T14:35:02.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:02.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976983+0000", 2024-04-24T14:35:02.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069735600000000002 2024-04-24T14:35:02.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:02.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992413+0000", 2024-04-24T14:35:02.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015429841999999999 2024-04-24T14:35:02.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:02.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992457+0000", 2024-04-24T14:35:02.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4169999999999999e-05 2024-04-24T14:35:02.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992508+0000", 2024-04-24T14:35:02.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0995000000000002e-05 2024-04-24T14:35:02.556 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.556 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.556 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.556 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:113 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:02.556 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:02.556 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.498983129000003, 2024-04-24T14:35:02.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014793863000000001, 2024-04-24T14:35:02.557 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.557 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:02.557 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.557 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:02.557 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:02.557 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 113 2024-04-24T14:35:02.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.557 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:02.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:02.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980152+0000", 2024-04-24T14:35:02.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:02.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980156+0000", 2024-04-24T14:35:02.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.619e-06 2024-04-24T14:35:02.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980163+0000", 2024-04-24T14:35:02.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7070000000000003e-06 2024-04-24T14:35:02.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980173+0000", 2024-04-24T14:35:02.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0957e-05 2024-04-24T14:35:02.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983381+0000", 2024-04-24T14:35:02.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032079930000000001 2024-04-24T14:35:02.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983628+0000", 2024-04-24T14:35:02.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024665700000000001 2024-04-24T14:35:02.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:02.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983964+0000", 2024-04-24T14:35:02.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000336332 2024-04-24T14:35:02.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:02.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992694+0000", 2024-04-24T14:35:02.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087297009999999994 2024-04-24T14:35:02.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:02.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994878+0000", 2024-04-24T14:35:02.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021841439999999998 2024-04-24T14:35:02.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:02.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994917+0000", 2024-04-24T14:35:02.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8544999999999998e-05 2024-04-24T14:35:02.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994947+0000", 2024-04-24T14:35:02.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9705000000000001e-05 2024-04-24T14:35:02.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.562 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.562 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:130 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:02.562 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:02.563 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.473289694000002, 2024-04-24T14:35:02.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014646289999999999, 2024-04-24T14:35:02.563 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.563 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:02.563 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.563 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:02.563 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:02.563 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 130 2024-04-24T14:35:02.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.563 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005844+0000", 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005853+0000", 2024-04-24T14:35:02.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1319999999999994e-06 2024-04-24T14:35:02.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005870+0000", 2024-04-24T14:35:02.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7448999999999998e-05 2024-04-24T14:35:02.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005876+0000", 2024-04-24T14:35:02.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3319999999999997e-06 2024-04-24T14:35:02.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005913+0000", 2024-04-24T14:35:02.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6835999999999998e-05 2024-04-24T14:35:02.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006017+0000", 2024-04-24T14:35:02.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010358500000000001 2024-04-24T14:35:02.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:02.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006139+0000", 2024-04-24T14:35:02.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001227 2024-04-24T14:35:02.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.010194+0000", 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040549380000000001 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020365+0000", 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010170983 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020428+0000", 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2457000000000002e-05 2024-04-24T14:35:02.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020492+0000", 2024-04-24T14:35:02.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4592999999999996e-05 2024-04-24T14:35:02.568 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.568 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.568 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.568 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:155 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1601536~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:02.568 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:02.568 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.439628906999999, 2024-04-24T14:35:02.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015144391, 2024-04-24T14:35:02.569 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.569 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:02.569 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.569 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:02.569 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:02.569 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 155 2024-04-24T14:35:02.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.569 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:02.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:02.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039506+0000", 2024-04-24T14:35:02.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:02.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039766+0000", 2024-04-24T14:35:02.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025994400000000002 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039780+0000", 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3366e-05 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039787+0000", 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4359999999999996e-06 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.050886+0000", 2024-04-24T14:35:02.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011098993 2024-04-24T14:35:02.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051026+0000", 2024-04-24T14:35:02.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001395 2024-04-24T14:35:02.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:02.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051211+0000", 2024-04-24T14:35:02.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018547099999999999 2024-04-24T14:35:02.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:02.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054490+0000", 2024-04-24T14:35:02.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032785499999999999 2024-04-24T14:35:02.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:02.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054505+0000", 2024-04-24T14:35:02.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5906999999999999e-05 2024-04-24T14:35:02.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:02.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054594+0000", 2024-04-24T14:35:02.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8817e-05 2024-04-24T14:35:02.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054651+0000", 2024-04-24T14:35:02.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7040999999999998e-05 2024-04-24T14:35:02.574 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.574 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.574 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.574 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:156 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1732608~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:02.574 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:02.574 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.439338200999998, 2024-04-24T14:35:02.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016895194999999998, 2024-04-24T14:35:02.575 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.575 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:02.575 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.575 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:02.575 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:02.575 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 156 2024-04-24T14:35:02.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.575 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:02.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:02.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039797+0000", 2024-04-24T14:35:02.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:02.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040065+0000", 2024-04-24T14:35:02.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000268361 2024-04-24T14:35:02.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040086+0000", 2024-04-24T14:35:02.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1024000000000001e-05 2024-04-24T14:35:02.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040094+0000", 2024-04-24T14:35:02.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9379999999999992e-06 2024-04-24T14:35:02.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051936+0000", 2024-04-24T14:35:02.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011842518 2024-04-24T14:35:02.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052062+0000", 2024-04-24T14:35:02.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125482 2024-04-24T14:35:02.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:02.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052206+0000", 2024-04-24T14:35:02.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143953 2024-04-24T14:35:02.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:02.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054715+0000", 2024-04-24T14:35:02.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025088240000000002 2024-04-24T14:35:02.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:02.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056534+0000", 2024-04-24T14:35:02.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001819436 2024-04-24T14:35:02.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:02.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056610+0000", 2024-04-24T14:35:02.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5383000000000007e-05 2024-04-24T14:35:02.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056693+0000", 2024-04-24T14:35:02.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3283999999999997e-05 2024-04-24T14:35:02.580 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.580 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.580 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.580 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:157 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1863680~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:02.580 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:02.580 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.439030821999999, 2024-04-24T14:35:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016763674999999999, 2024-04-24T14:35:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157 2024-04-24T14:35:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:02.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040104+0000", 2024-04-24T14:35:02.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:02.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040244+0000", 2024-04-24T14:35:02.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014035499999999999 2024-04-24T14:35:02.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040255+0000", 2024-04-24T14:35:02.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0135999999999999e-05 2024-04-24T14:35:02.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040260+0000", 2024-04-24T14:35:02.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7919999999999998e-06 2024-04-24T14:35:02.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053406+0000", 2024-04-24T14:35:02.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013146023999999999 2024-04-24T14:35:02.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053531+0000", 2024-04-24T14:35:02.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012471800000000001 2024-04-24T14:35:02.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:02.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053715+0000", 2024-04-24T14:35:02.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018395099999999999 2024-04-24T14:35:02.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:02.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056711+0000", 2024-04-24T14:35:02.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029956520000000001 2024-04-24T14:35:02.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:02.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056783+0000", 2024-04-24T14:35:02.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2010999999999999e-05 2024-04-24T14:35:02.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:02.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056831+0000", 2024-04-24T14:35:02.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8189999999999998e-05 2024-04-24T14:35:02.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056869+0000", 2024-04-24T14:35:02.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7812000000000003e-05 2024-04-24T14:35:02.586 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.586 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.586 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.586 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:158 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1994752~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:02.586 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:02.587 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.438868894999999, 2024-04-24T14:35:02.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016845334999999999, 2024-04-24T14:35:02.587 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.587 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:02.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:02.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:02.587 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2024-04-24T14:35:02.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.587 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:02.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:02.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040266+0000", 2024-04-24T14:35:02.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:02.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040379+0000", 2024-04-24T14:35:02.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112728 2024-04-24T14:35:02.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040387+0000", 2024-04-24T14:35:02.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8499999999999994e-06 2024-04-24T14:35:02.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040391+0000", 2024-04-24T14:35:02.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2719999999999999e-06 2024-04-24T14:35:02.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054803+0000", 2024-04-24T14:35:02.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014411548 2024-04-24T14:35:02.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054982+0000", 2024-04-24T14:35:02.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017941699999999999 2024-04-24T14:35:02.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:02.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.055167+0000", 2024-04-24T14:35:02.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018542100000000001 2024-04-24T14:35:02.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:02.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056718+0000", 2024-04-24T14:35:02.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015505670000000001 2024-04-24T14:35:02.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:02.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057020+0000", 2024-04-24T14:35:02.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030227500000000002 2024-04-24T14:35:02.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:02.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057070+0000", 2024-04-24T14:35:02.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9611000000000003e-05 2024-04-24T14:35:02.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057112+0000", 2024-04-24T14:35:02.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2373999999999998e-05 2024-04-24T14:35:02.592 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.592 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.592 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.592 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:159 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:02.593 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:02.593 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.433433907000001, 2024-04-24T14:35:02.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015930650000000001, 2024-04-24T14:35:02.593 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.593 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.593 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.593 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:02.593 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:02.593 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2024-04-24T14:35:02.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.593 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:02.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:02.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045701+0000", 2024-04-24T14:35:02.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:02.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045707+0000", 2024-04-24T14:35:02.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.562e-06 2024-04-24T14:35:02.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045728+0000", 2024-04-24T14:35:02.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1109999999999999e-05 2024-04-24T14:35:02.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045738+0000", 2024-04-24T14:35:02.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0016e-05 2024-04-24T14:35:02.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056183+0000", 2024-04-24T14:35:02.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010444836000000001 2024-04-24T14:35:02.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:02.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056470+0000", 2024-04-24T14:35:02.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028696899999999998 2024-04-24T14:35:02.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057160+0000", 2024-04-24T14:35:02.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069004699999999995 2024-04-24T14:35:02.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057291+0000", 2024-04-24T14:35:02.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013096200000000001 2024-04-24T14:35:02.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.061633+0000", 2024-04-24T14:35:02.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004341414 2024-04-24T14:35:02.597 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.597 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.597 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.598 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.598 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.598 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:02.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:35:02.599 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:02.604+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:02.616 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-04-24T14:35:02.618+0000 7ff9c25de700 -1 mon.a@0(leader).osd e106 definitely_dead 0 2024-04-24T14:35:02.697 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:02.697 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:02.697 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:35:02.697 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:02.700 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:02.704+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:02.706 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:35:02.801 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:02.804+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:02.853 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:02.854 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:02.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:35:02.902 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:02.906+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:02.962 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.494524455999994, 2024-04-24T14:35:02.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:35:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:35:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:35:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:35:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:35:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:35:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:35:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:35:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:35:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:35:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:35:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:35:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:35:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:35:02.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.119167752999999, 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.967 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:35:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:02.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:35:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:35:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:35:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:35:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:35:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:35:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:35:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:35:02.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:35:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:35:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:35:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:35:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:35:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.971 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.117739599000004, 2024-04-24T14:35:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:35:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.972 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:35:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:35:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:35:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:35:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:35:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:35:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:35:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:35:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:35:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:35:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:35:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:35:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:35:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:35:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.117712510000004, 2024-04-24T14:35:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:35:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:35:02.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:35:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:35:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:35:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:35:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:35:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:35:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:35:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:35:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:35:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:35:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:35:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:35:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:35:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.981 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.117688543, 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:35:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:35:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:35:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:35:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:35:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:35:02.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:35:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:35:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:35:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:35:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:35:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.103704563999997, 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:35:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:35:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:35:02.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:35:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:35:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:35:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:35:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:35:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:35:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:35:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:35:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:35:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:35:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:35:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:35:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:35:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:35:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:35:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:35:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:35:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:35:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:35:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.992 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.103588252999998, 2024-04-24T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:02.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:35:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:35:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:35:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:35:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:35:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:35:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:35:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:35:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:35:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:35:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:35:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:35:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:35:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:35:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:35:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:35:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:35:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:35:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:35:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:35:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:35:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.103540594999998, 2024-04-24T14:35:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:35:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:35:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:35:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:35:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:35:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:35:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:35:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:35:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:35:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:35:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:35:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:35:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:35:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:35:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:35:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:35:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:35:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:35:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:35:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:35:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:35:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:35:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:35:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.099503296999998, 2024-04-24T14:35:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:35:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:35:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:35:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:35:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:35:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:35:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:35:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:35:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:35:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:35:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:35:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:35:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:35:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:35:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:35:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:35:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:35:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:35:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:35:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:35:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:35:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:35:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:35:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.099439537999999, 2024-04-24T14:35:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:35:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:35:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:35:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:35:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:35:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:35:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:35:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:35:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:35:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:35:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:35:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:35:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:35:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:35:03.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:35:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:35:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:35:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:35:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:35:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:35:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:35:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:35:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:35:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.099414048, 2024-04-24T14:35:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:35:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:35:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:03.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:35:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:35:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:35:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:35:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:35:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:35:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:35:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:35:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:35:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:35:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:35:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:35:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:35:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:35:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:35:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:35:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:35:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:35:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:35:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:35:03.024 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.099390298000003, 2024-04-24T14:35:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:35:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:03.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:35:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:35:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:35:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:35:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:35:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:35:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:35:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:35:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:35:03.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:35:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:35:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:35:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:35:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:35:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:35:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:35:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:35:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:35:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:35:03.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:35:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:35:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:35:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.031 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.099351784999996, 2024-04-24T14:35:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:35:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:35:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:35:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:35:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:35:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:35:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:35:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:35:03.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:35:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:35:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:35:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:35:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:35:03.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:35:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:35:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:35:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:35:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:35:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:35:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:35:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:35:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:35:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:35:03.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.099323588000004, 2024-04-24T14:35:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:35:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:03.038 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:35:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:35:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:03.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:35:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:35:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:35:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:35:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:35:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:35:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:35:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:35:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:35:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:35:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:35:03.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:35:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:35:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:35:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:35:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:35:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:35:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:35:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:35:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:35:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.044 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.099302131000002, 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:03.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:35:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:35:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:35:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:35:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:35:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:35:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:35:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:35:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:35:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:35:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:35:03.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:35:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:35:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:35:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:35:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:35:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:35:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:35:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:35:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:35:03.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:35:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.099283288999999, 2024-04-24T14:35:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:35:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:03.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:35:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:35:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:35:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:35:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:03.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:35:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:35:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:35:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:35:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:35:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:35:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:35:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:35:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:35:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:35:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:35:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:35:03.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:35:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:35:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:35:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:35:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:35:03.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:35:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.886451139999998, 2024-04-24T14:35:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:35:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:03.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:35:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:35:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:03.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:35:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:35:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:35:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:35:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:35:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:35:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:35:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:35:03.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:35:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:35:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:35:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:35:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:35:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:35:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:35:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:35:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:35:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:35:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:35:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:35:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:35:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:39 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1753088~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.946678724000002, 2024-04-24T14:35:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01139394, 2024-04-24T14:35:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 39 2024-04-24T14:35:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018967+0000", 2024-04-24T14:35:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019242+0000", 2024-04-24T14:35:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027504500000000002 2024-04-24T14:35:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019274+0000", 2024-04-24T14:35:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2472999999999999e-05 2024-04-24T14:35:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019288+0000", 2024-04-24T14:35:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3283e-05 2024-04-24T14:35:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026753+0000", 2024-04-24T14:35:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074651120000000003 2024-04-24T14:35:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026971+0000", 2024-04-24T14:35:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021810800000000001 2024-04-24T14:35:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.027211+0000", 2024-04-24T14:35:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023975 2024-04-24T14:35:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030200+0000", 2024-04-24T14:35:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029890709999999998 2024-04-24T14:35:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030252+0000", 2024-04-24T14:35:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1926999999999999e-05 2024-04-24T14:35:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030312+0000", 2024-04-24T14:35:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0602e-05 2024-04-24T14:35:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030363+0000", 2024-04-24T14:35:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0342e-05 2024-04-24T14:35:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.069 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:40 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1884160~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.946342749999999, 2024-04-24T14:35:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011199064, 2024-04-24T14:35:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 40 2024-04-24T14:35:03.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:03.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019303+0000", 2024-04-24T14:35:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019729+0000", 2024-04-24T14:35:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042620899999999998 2024-04-24T14:35:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:03.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019769+0000", 2024-04-24T14:35:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9847e-05 2024-04-24T14:35:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019797+0000", 2024-04-24T14:35:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8347000000000002e-05 2024-04-24T14:35:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028299+0000", 2024-04-24T14:35:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085010629999999997 2024-04-24T14:35:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028385+0000", 2024-04-24T14:35:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6824999999999996e-05 2024-04-24T14:35:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028510+0000", 2024-04-24T14:35:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012483000000000001 2024-04-24T14:35:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030410+0000", 2024-04-24T14:35:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001899612 2024-04-24T14:35:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030416+0000", 2024-04-24T14:35:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7699999999999998e-06 2024-04-24T14:35:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030461+0000", 2024-04-24T14:35:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5157999999999997e-05 2024-04-24T14:35:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030504+0000", 2024-04-24T14:35:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2988999999999998e-05 2024-04-24T14:35:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:41 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 2015232~81920] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:03.076 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.945834043000001, 2024-04-24T14:35:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012345257, 2024-04-24T14:35:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 41 2024-04-24T14:35:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019812+0000", 2024-04-24T14:35:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019975+0000", 2024-04-24T14:35:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016325700000000001 2024-04-24T14:35:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020025+0000", 2024-04-24T14:35:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9484999999999998e-05 2024-04-24T14:35:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020035+0000", 2024-04-24T14:35:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0444e-05 2024-04-24T14:35:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029150+0000", 2024-04-24T14:35:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091150160000000001 2024-04-24T14:35:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029352+0000", 2024-04-24T14:35:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020166499999999999 2024-04-24T14:35:03.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029611+0000", 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025949600000000001 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030518+0000", 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090633300000000005 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032088+0000", 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015703290000000001 2024-04-24T14:35:03.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032130+0000", 2024-04-24T14:35:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2273e-05 2024-04-24T14:35:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032159+0000", 2024-04-24T14:35:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8387000000000001e-05 2024-04-24T14:35:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.082 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:03.083 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:03.083 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:03.084 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:35:03.085 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:03.090+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:03.185 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:03.190+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:03.192 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:03.192 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:35:03.192 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:03.202 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:35:03.287 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:03.288+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:03.293 INFO:teuthology.orchestra.run.smithi090.stderr:marked in osd.5. 2024-04-24T14:35:03.307 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-04-24T14:35:03.308 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:03.308 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:03.308 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:35:03.309 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:35:03.309 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:03.317 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:35:03.387 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:03.392+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:03.412 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:03.413 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:35:03.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:35:03.413 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:35:03.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.413 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:03.413 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:03.413 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 160.78954688300001, 2024-04-24T14:35:03.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:35:03.413 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.413 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.413 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.413 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:03.413 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:03.414 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:35:03.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.414 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:03.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:03.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:35:03.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:03.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:35:03.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:35:03.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:35:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:35:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:35:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:35:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:35:03.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:35:03.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:35:03.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:35:03.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:03.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:35:03.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:35:03.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:35:03.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:35:03.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:03.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:35:03.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:35:03.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:35:03.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:35:03.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:35:03.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:35:03.419 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.419 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.419 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.420 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:03.420 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:03.420 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 160.78936979599999, 2024-04-24T14:35:03.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:35:03.420 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.420 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:03.420 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:03.420 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:35:03.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.420 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:35:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:35:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:35:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:35:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:35:03.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:35:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:35:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:35:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:35:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:35:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:35:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:35:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:35:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:35:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:35:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:35:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:35:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:35:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:35:03.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:35:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:35:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:03.426 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 160.789265882, 2024-04-24T14:35:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:35:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:35:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:35:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:03.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:35:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:35:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:35:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:35:03.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:35:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:35:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:35:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:35:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:35:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:35:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:35:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:35:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:35:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:35:03.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:35:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:35:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:35:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:35:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:35:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:35:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 160.788877431, 2024-04-24T14:35:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:35:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:35:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:03.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:35:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:03.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:35:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:35:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:35:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:35:03.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:35:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:35:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:35:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:35:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:35:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:35:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:35:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:35:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:35:03.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:35:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:35:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:35:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:35:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:35:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:35:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:35:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 160.78879221899999, 2024-04-24T14:35:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:35:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.440 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:35:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:35:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:35:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:35:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:35:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:35:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:35:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:35:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:35:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:35:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:35:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:35:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:35:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:35:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:35:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:35:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:35:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:35:03.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:35:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:35:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:35:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:35:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.446 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 160.78855533800001, 2024-04-24T14:35:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:35:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.447 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:35:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:35:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:35:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:35:03.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:35:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:35:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:35:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:35:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:35:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:35:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:35:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:35:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:35:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:35:03.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:35:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:35:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:35:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:35:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:35:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:35:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:35:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:35:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 160.78847381599999, 2024-04-24T14:35:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:35:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:35:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:35:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:35:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:35:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:35:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:35:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:35:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:35:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:35:03.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:35:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:35:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:35:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:35:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:35:03.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:35:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:35:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:35:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:35:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:35:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:35:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:35:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:35:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 160.35656146900001, 2024-04-24T14:35:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:35:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:35:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.461 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:35:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:35:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:35:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:35:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:35:03.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:35:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:35:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:35:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:35:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:35:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:35:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:35:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:35:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:35:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:35:03.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:35:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:35:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:35:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:35:03.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:35:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:35:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 160.35653316899999, 2024-04-24T14:35:03.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:35:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:35:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:35:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:35:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:35:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:35:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:35:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:35:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:35:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:35:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:35:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:35:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:35:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:35:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:35:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:35:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:35:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:35:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:35:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:35:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:35:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:35:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:35:03.473 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 160.35644546500001, 2024-04-24T14:35:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:35:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.474 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:35:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:35:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:35:03.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:35:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:35:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:35:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:35:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:35:03.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:35:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:35:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:35:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:35:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:35:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:35:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:35:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:35:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:35:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:35:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:35:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:35:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:35:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:35:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.480 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 160.356322664, 2024-04-24T14:35:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:35:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:35:03.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:03.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:35:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:35:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:35:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:35:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:35:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:35:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:35:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:35:03.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:35:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:35:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:35:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:35:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:35:03.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:35:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:35:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:35:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:35:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:35:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:35:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:35:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:35:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:35:03.487 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.050078103999994, 2024-04-24T14:35:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:35:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:35:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:35:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:35:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:35:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:03.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:35:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:35:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:35:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:35:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:35:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:35:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:35:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:35:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:35:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:35:03.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:35:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:35:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:35:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:35:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:35:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:35:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:35:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 83.355954104999995, 2024-04-24T14:35:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:35:03.493 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:35:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:35:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:35:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:35:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.495 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:35:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:35:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:35:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:35:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:35:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:35:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:35:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:35:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:35:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:35:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:35:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:35:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:35:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:35:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:35:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:35:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:35:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:35:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:35:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:35:03.499 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.396594139000001, 2024-04-24T14:35:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:35:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:03.500 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:35:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:35:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:35:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:35:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:35:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:35:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:35:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:35:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:35:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:35:03.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:35:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:35:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:35:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:35:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:35:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:35:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:35:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:35:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:35:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:35:03.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:35:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:35:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.396513584000004, 2024-04-24T14:35:03.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:35:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:35:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:35:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:35:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:35:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:35:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:35:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:35:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:35:03.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:35:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:35:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:35:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:35:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:35:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:35:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:35:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:35:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:35:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:35:03.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:35:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:35:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:35:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:35:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.396465855000002, 2024-04-24T14:35:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:35:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:35:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.513 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:35:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:35:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:35:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:35:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:35:03.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:35:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:35:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:35:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:35:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:35:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:35:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:35:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:35:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:35:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:35:03.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:35:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:35:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:35:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:35:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:35:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:35:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.396438091999997, 2024-04-24T14:35:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:35:03.519 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.520 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.520 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.520 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:03.520 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:03.520 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:35:03.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.520 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:03.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:03.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:35:03.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:35:03.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:35:03.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:35:03.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:35:03.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:35:03.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:35:03.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:35:03.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:35:03.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:35:03.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:35:03.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:35:03.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:35:03.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:35:03.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:35:03.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:35:03.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:35:03.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:35:03.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:35:03.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:35:03.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:35:03.525 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.525 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.525 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.526 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:03.526 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:03.526 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.396398657000006, 2024-04-24T14:35:03.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:35:03.526 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.526 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.526 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.526 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:03.526 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:03.526 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:35:03.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.527 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:03.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:03.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:35:03.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:35:03.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:35:03.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:35:03.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:35:03.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:35:03.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:35:03.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:35:03.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:35:03.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:35:03.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:35:03.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:35:03.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:35:03.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:35:03.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:35:03.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:35:03.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:35:03.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:35:03.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:35:03.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:35:03.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:35:03.532 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.532 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.532 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.532 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:03.532 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:03.532 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.396373672999999, 2024-04-24T14:35:03.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:35:03.533 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.533 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.533 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.533 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:03.533 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:03.533 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:35:03.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.533 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:03.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:03.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:35:03.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:35:03.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:35:03.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:35:03.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:35:03.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:35:03.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:35:03.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:35:03.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:35:03.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:35:03.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:35:03.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:35:03.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:35:03.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:35:03.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:35:03.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:35:03.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:35:03.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:35:03.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:35:03.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:35:03.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:35:03.539 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.539 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.539 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.540 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:03.540 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:03.540 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 64.396348013999997, 2024-04-24T14:35:03.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:35:03.540 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.540 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.540 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.540 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:03.540 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:03.540 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:35:03.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.541 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:03.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:03.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:35:03.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:35:03.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:35:03.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:35:03.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:35:03.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:35:03.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:35:03.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:35:03.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:35:03.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:03.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:35:03.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:35:03.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:35:03.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:35:03.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:03.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:35:03.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:35:03.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:35:03.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:35:03.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:35:03.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:35:03.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:35:03.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:35:03.546 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.546 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.546 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.546 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.546 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.546 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:03.548 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:35:03.548 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:03.552+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:03.648 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:03.650+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:03.653 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:03.653 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:03.653 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:35:03.653 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:03.663 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:35:03.749 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:03.750+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:03.758 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:03.758 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:03.758 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:35:03.759 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:35:03.759 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:03.768 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:35:03.849 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:03.852+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:03.868 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:03.868 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:35:03.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:35:03.868 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:35:03.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.868 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:03.868 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:03.868 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.208593844999996, 2024-04-24T14:35:03.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:35:03.869 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.869 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.869 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.869 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:03.869 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.869 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:35:03.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.869 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:03.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:03.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:03.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:35:03.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:35:03.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:35:03.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:35:03.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:35:03.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:35:03.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:35:03.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:35:03.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:35:03.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:35:03.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.873 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.873 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:35:03.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:35:03.873 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.873 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.873 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.873 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:03.873 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:35:03.873 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.205559319000002, 2024-04-24T14:35:03.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:35:03.874 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.874 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.874 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.874 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:03.874 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.874 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:35:03.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.874 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:35:03.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:35:03.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:35:03.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:35:03.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:35:03.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:35:03.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:35:03.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:35:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:35:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:35:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:35:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:35:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:35:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.877 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:35:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:35:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.205535132999998, 2024-04-24T14:35:03.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:35:03.879 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.879 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.879 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.879 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:03.879 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.879 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:35:03.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.879 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:03.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.880 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:03.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.880 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:03.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.880 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:35:03.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:35:03.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:35:03.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:35:03.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:35:03.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:35:03.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:35:03.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:35:03.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:35:03.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:35:03.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:35:03.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:35:03.883 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.883 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.883 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.883 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:03.883 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:35:03.883 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.205517748999995, 2024-04-24T14:35:03.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:35:03.883 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.883 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.883 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.883 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:03.883 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.884 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:35:03.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.884 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:35:03.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:35:03.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:35:03.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.885 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:35:03.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:35:03.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:35:03.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:35:03.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:35:03.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:35:03.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:35:03.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:35:03.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:35:03.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:35:03.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:35:03.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:35:03.887 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.888 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.888 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.888 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:03.888 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:35:03.888 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.191081656999998, 2024-04-24T14:35:03.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:35:03.888 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.888 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.888 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.888 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:03.888 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.889 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:35:03.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.889 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:35:03.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:35:03.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:35:03.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:35:03.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:35:03.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:35:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:35:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:35:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:35:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:35:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:35:03.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:35:03.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:35:03.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:35:03.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:35:03.892 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.892 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.893 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.893 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:03.893 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:35:03.893 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.191049358000001, 2024-04-24T14:35:03.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:35:03.893 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.893 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.893 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.893 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:03.893 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.893 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:35:03.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.894 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:35:03.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.894 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.894 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:35:03.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:35:03.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:35:03.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:35:03.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:35:03.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:35:03.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:35:03.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:35:03.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:35:03.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:35:03.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:35:03.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:35:03.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:35:03.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:35:03.897 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.897 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.897 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.898 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:03.898 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:35:03.898 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.176797081000004, 2024-04-24T14:35:03.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:35:03.898 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.898 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.898 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.898 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:03.898 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.898 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:35:03.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.899 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:35:03.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:35:03.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:35:03.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:35:03.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:35:03.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:35:03.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:35:03.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:35:03.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:35:03.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:35:03.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:35:03.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:35:03.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:35:03.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.902 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:35:03.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:35:03.902 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.902 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.902 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.903 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:03.903 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:03.903 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.176767980999998, 2024-04-24T14:35:03.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:35:03.903 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.903 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.903 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.903 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:03.903 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.903 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:35:03.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.904 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:03.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:03.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:03.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:35:03.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:35:03.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:35:03.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:35:03.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:35:03.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:35:03.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:35:03.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:35:03.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:35:03.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:35:03.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:35:03.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:35:03.907 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.907 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.907 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.907 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:03.908 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:03.908 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.176742051000005, 2024-04-24T14:35:03.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:35:03.908 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.908 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.908 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.908 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:03.908 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.908 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:35:03.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.908 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:03.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:03.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:03.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:35:03.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:35:03.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:35:03.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:35:03.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:35:03.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:35:03.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:35:03.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:35:03.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:35:03.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:35:03.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.912 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.912 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:35:03.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:35:03.912 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.912 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.912 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.912 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:35:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.161021586999993, 2024-04-24T14:35:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:35:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:35:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:35:03.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:35:03.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:35:03.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:35:03.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:35:03.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:35:03.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:35:03.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:35:03.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:35:03.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:35:03.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:35:03.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:35:03.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:35:03.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:35:03.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:35:03.917 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.917 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.917 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.917 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:03.918 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:35:03.918 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.160989654999995, 2024-04-24T14:35:03.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:35:03.918 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.918 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:35:03.918 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.918 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:03.918 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.918 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:35:03.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.918 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:35:03.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:35:03.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:35:03.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:35:03.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:35:03.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:35:03.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:35:03.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:35:03.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:35:03.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:35:03.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:35:03.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:35:03.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:35:03.922 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.922 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.922 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.922 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:03.922 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:35:03.922 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.160922138999993, 2024-04-24T14:35:03.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:35:03.922 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.922 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:35:03.922 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.923 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:03.923 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:03.923 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:35:03.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.923 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:35:03.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:35:03.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:35:03.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:03.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:35:03.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:35:03.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:35:03.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:35:03.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:35:03.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:35:03.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:35:03.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:35:03.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.926 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.926 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:35:03.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:35:03.926 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.926 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.926 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.926 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.927 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:35:03.927 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:03.927 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.103509255999995, 2024-04-24T14:35:03.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:35:03.927 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.927 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.927 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:03.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:03.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:03.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:03.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:35:03.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:35:03.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:35:03.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:35:03.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:35:03.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:35:03.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:35:03.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:35:03.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:35:03.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:35:03.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:35:03.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:35:03.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:35:03.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:35:03.931 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.931 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.931 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.932 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:103 3.1 e87/74 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head v 87'261, mlcod=87'258)", 2024-04-24T14:35:03.932 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:35:03.932 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.912228052000003, 2024-04-24T14:35:03.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019745728000000001, 2024-04-24T14:35:03.932 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.932 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.932 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:35:03.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:35:03.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960679+0000", 2024-04-24T14:35:03.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:03.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960692+0000", 2024-04-24T14:35:03.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3709999999999999e-05 2024-04-24T14:35:03.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960722+0000", 2024-04-24T14:35:03.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9791999999999999e-05 2024-04-24T14:35:03.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960733+0000", 2024-04-24T14:35:03.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.128e-05 2024-04-24T14:35:03.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960857+0000", 2024-04-24T14:35:03.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000123981 2024-04-24T14:35:03.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960926+0000", 2024-04-24T14:35:03.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8881999999999995e-05 2024-04-24T14:35:03.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980353+0000", 2024-04-24T14:35:03.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019426853000000001 2024-04-24T14:35:03.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980426+0000", 2024-04-24T14:35:03.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2803000000000001e-05 2024-04-24T14:35:03.936 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.936 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.936 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.937 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:105 3.1 e87/74 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head v 87'263, mlcod=87'258)", 2024-04-24T14:35:03.937 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:35:03.937 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.903101518, 2024-04-24T14:35:03.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014864743999999999, 2024-04-24T14:35:03.937 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.937 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.937 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:35:03.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:35:03.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969805+0000", 2024-04-24T14:35:03.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:03.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969820+0000", 2024-04-24T14:35:03.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5116999999999999e-05 2024-04-24T14:35:03.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969852+0000", 2024-04-24T14:35:03.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2135999999999999e-05 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969864+0000", 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.189e-05 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972122+0000", 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022580389999999999 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972165+0000", 2024-04-24T14:35:03.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2429000000000003e-05 2024-04-24T14:35:03.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984619+0000", 2024-04-24T14:35:03.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012454735 2024-04-24T14:35:03.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984672+0000", 2024-04-24T14:35:03.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2077999999999998e-05 2024-04-24T14:35:03.941 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.941 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.941 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.942 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:112 3.1 e87/74 3:a983798d:::rbd_data.12ff64619e2d.0000000000000006:head v 87'265, mlcod=87'258)", 2024-04-24T14:35:03.942 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:35:03.942 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.892762681999997, 2024-04-24T14:35:03.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012821978, 2024-04-24T14:35:03.942 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.942 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.942 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980144+0000", 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980159+0000", 2024-04-24T14:35:03.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4769999999999999e-05 2024-04-24T14:35:03.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980189+0000", 2024-04-24T14:35:03.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0419000000000001e-05 2024-04-24T14:35:03.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980213+0000", 2024-04-24T14:35:03.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4036000000000002e-05 2024-04-24T14:35:03.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980452+0000", 2024-04-24T14:35:03.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00023921500000000001 2024-04-24T14:35:03.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980468+0000", 2024-04-24T14:35:03.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5843999999999999e-05 2024-04-24T14:35:03.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992932+0000", 2024-04-24T14:35:03.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012463314 2024-04-24T14:35:03.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992968+0000", 2024-04-24T14:35:03.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6075999999999998e-05 2024-04-24T14:35:03.946 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.946 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.946 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.947 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:113 3.1 e87/74 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head v 87'267, mlcod=87'258)", 2024-04-24T14:35:03.947 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:35:03.947 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.889227957999999, 2024-04-24T14:35:03.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010561022999999999, 2024-04-24T14:35:03.947 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.947 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.947 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:35:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:35:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983679+0000", 2024-04-24T14:35:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983699+0000", 2024-04-24T14:35:03.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0392000000000001e-05 2024-04-24T14:35:03.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983737+0000", 2024-04-24T14:35:03.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8340000000000002e-05 2024-04-24T14:35:03.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983756+0000", 2024-04-24T14:35:03.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8402000000000001e-05 2024-04-24T14:35:03.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984706+0000", 2024-04-24T14:35:03.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095024400000000005 2024-04-24T14:35:03.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984741+0000", 2024-04-24T14:35:03.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4721000000000002e-05 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994211+0000", 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094708559999999997 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994241+0000", 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9941e-05 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.952 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:115 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:03.952 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:35:03.952 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.888773921000002, 2024-04-24T14:35:03.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010347242, 2024-04-24T14:35:03.952 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.952 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.952 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.952 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:35:03.952 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:03.952 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:35:03.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.953 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:35:03.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:35:03.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984132+0000", 2024-04-24T14:35:03.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:03.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984142+0000", 2024-04-24T14:35:03.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7370000000000001e-06 2024-04-24T14:35:03.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984154+0000", 2024-04-24T14:35:03.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2288e-05 2024-04-24T14:35:03.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984163+0000", 2024-04-24T14:35:03.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3749999999999992e-06 2024-04-24T14:35:03.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984246+0000", 2024-04-24T14:35:03.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2962999999999998e-05 2024-04-24T14:35:03.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984473+0000", 2024-04-24T14:35:03.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00022696399999999999 2024-04-24T14:35:03.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:03.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984771+0000", 2024-04-24T14:35:03.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00029739499999999999 2024-04-24T14:35:03.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992998+0000", 2024-04-24T14:35:03.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082275750000000009 2024-04-24T14:35:03.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:03.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994299+0000", 2024-04-24T14:35:03.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013005289999999999 2024-04-24T14:35:03.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994403+0000", 2024-04-24T14:35:03.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010376499999999999 2024-04-24T14:35:03.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994482+0000", 2024-04-24T14:35:03.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9013999999999999e-05 2024-04-24T14:35:03.958 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.958 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.958 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.958 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:128 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:03.959 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:35:03.959 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.867582992999999, 2024-04-24T14:35:03.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011784144, 2024-04-24T14:35:03.959 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.959 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.959 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:03.959 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:35:03.959 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:03.959 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 128 2024-04-24T14:35:03.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.959 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:35:03.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:35:03.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005324+0000", 2024-04-24T14:35:03.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:03.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005332+0000", 2024-04-24T14:35:03.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2989999999999997e-06 2024-04-24T14:35:03.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005354+0000", 2024-04-24T14:35:03.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1705999999999999e-05 2024-04-24T14:35:03.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005362+0000", 2024-04-24T14:35:03.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5599999999999996e-06 2024-04-24T14:35:03.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005418+0000", 2024-04-24T14:35:03.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6685999999999999e-05 2024-04-24T14:35:03.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005535+0000", 2024-04-24T14:35:03.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117008 2024-04-24T14:35:03.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:03.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005732+0000", 2024-04-24T14:35:03.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019718700000000001 2024-04-24T14:35:03.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:03.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016440+0000", 2024-04-24T14:35:03.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010707787999999999 2024-04-24T14:35:03.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:03.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016913+0000", 2024-04-24T14:35:03.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047270400000000001 2024-04-24T14:35:03.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017047+0000", 2024-04-24T14:35:03.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013434699999999999 2024-04-24T14:35:03.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017109+0000", 2024-04-24T14:35:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2192000000000004e-05 2024-04-24T14:35:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:130 3.1 e87/74 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 87'269, mlcod=87'265)", 2024-04-24T14:35:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:35:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 36.867130232999997, 2024-04-24T14:35:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013945631999999999, 2024-04-24T14:35:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:35:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:03.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:35:03.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:03.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:03.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005777+0000", 2024-04-24T14:35:03.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:03.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:03.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005791+0000", 2024-04-24T14:35:03.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3743999999999999e-05 2024-04-24T14:35:03.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:03.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005803+0000", 2024-04-24T14:35:03.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2330999999999999e-05 2024-04-24T14:35:03.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:03.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005810+0000", 2024-04-24T14:35:03.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8990000000000004e-06 2024-04-24T14:35:03.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:03.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005888+0000", 2024-04-24T14:35:03.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7872000000000003e-05 2024-04-24T14:35:03.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:03.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005911+0000", 2024-04-24T14:35:03.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3156e-05 2024-04-24T14:35:03.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:03.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019642+0000", 2024-04-24T14:35:03.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013730456 2024-04-24T14:35:03.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:03.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:03.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:03.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019724+0000", 2024-04-24T14:35:03.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1981e-05 2024-04-24T14:35:03.970 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.970 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.970 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.970 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:03.970 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:03.970 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:03.972 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:35:03.973 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:03.976+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:04.073 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:04.076+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:04.078 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:04.079 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:35:04.079 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:04.088 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:35:04.174 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:04.178+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:04.208 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd1 2024-04-24T14:35:04.208 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:35:04.208 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:35:04.208 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:35:04.209 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:35:04.209 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd1'\'']/id' 2024-04-24T14:35:04.232 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:04.232 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:04.232 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:35:04.232 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:35:04.232 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:04.241 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:35:04.260 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:35:04.274 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:04.276+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:04.299 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=19651 2024-04-24T14:35:04.299 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 19651 2024-04-24T14:35:04.299 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 19651 -C rbd-nbd 2024-04-24T14:35:04.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T14:35:04.308 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:35:04.308 INFO:tasks.workunit.client.0.smithi173.stdout: 19651 ? 00:00:00 rbd-nbd 2024-04-24T14:35:04.309 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo blkdiscard --offset 0 --length 3584 /dev/nbd1 2024-04-24T14:35:04.309 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:35:04.309 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:35:04.311 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:35:04.311 INFO:tasks.workunit.client.0.smithi173.stderr:++ which blkdiscard 2024-04-24T14:35:04.312 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T14:35:04.313 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:35:04.313 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 3584 /dev/nbd1 2024-04-24T14:35:04.343 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-04-24T14:35:04.343 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:35:04.344 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:35:04.345 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:35:04.346 INFO:tasks.workunit.client.0.smithi173.stderr:++ which blkdiscard 2024-04-24T14:35:04.349 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T14:35:04.349 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:35:04.349 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-04-24T14:35:04.354 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:04.354 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:35:04.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:35:04.355 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:35:04.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.355 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:35:04.355 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:04.355 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 84.287130852999994, 2024-04-24T14:35:04.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:35:04.355 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.355 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:04.355 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:04.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:04.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:35:04.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:04.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:35:04.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:35:04.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:35:04.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:35:04.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:04.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:35:04.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:35:04.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:35:04.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:35:04.359 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.359 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.359 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.359 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:35:04.359 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:04.359 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 70.292316274000001, 2024-04-24T14:35:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:35:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:35:04.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:04.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:35:04.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:35:04.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:35:04.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:35:04.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:35:04.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:35:04.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:35:04.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:35:04.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:35:04.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:35:04.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:04.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:35:04.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:35:04.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:35:04.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:35:04.363 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.363 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.363 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.364 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:35:04.364 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:04.364 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 70.281387015999996, 2024-04-24T14:35:04.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:35:04.364 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.364 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:04.364 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:04.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:04.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:35:04.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:04.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:35:04.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:35:04.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:35:04.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:35:04.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:35:04.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:35:04.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:35:04.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:35:04.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:35:04.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:35:04.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:04.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:35:04.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:35:04.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:35:04.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:35:04.368 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.368 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.368 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.368 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:35:04.368 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:04.368 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 70.270430934999993, 2024-04-24T14:35:04.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:35:04.368 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.368 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:04.369 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:04.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:04.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:35:04.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:04.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:35:04.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:35:04.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:35:04.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:35:04.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:35:04.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:35:04.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:35:04.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:35:04.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:35:04.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:35:04.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:04.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:35:04.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:35:04.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:35:04.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:35:04.372 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.372 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.373 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.373 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:35:04.373 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:04.373 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 70.258861569999993, 2024-04-24T14:35:04.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:35:04.373 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.373 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:04.373 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:35:04.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:35:04.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:35:04.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:35:04.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:35:04.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:35:04.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:35:04.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:35:04.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:35:04.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:35:04.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:04.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:35:04.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:35:04.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:35:04.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:35:04.377 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.377 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.377 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.378 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:35:04.378 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:04.378 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 70.005741911000001, 2024-04-24T14:35:04.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:35:04.378 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.378 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:04.378 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:04.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:04.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:35:04.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:04.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:35:04.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:35:04.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:35:04.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:35:04.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:35:04.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:35:04.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:35:04.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:35:04.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:35:04.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:35:04.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:04.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:35:04.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:35:04.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:35:04.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:35:04.382 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.382 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.382 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.382 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:35:04.382 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:04.383 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.994996743000002, 2024-04-24T14:35:04.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:35:04.383 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.383 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:04.383 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:04.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:04.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:35:04.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:04.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:35:04.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:35:04.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:35:04.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:35:04.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:35:04.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:35:04.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:35:04.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:35:04.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:35:04.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:35:04.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:04.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:35:04.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:35:04.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:35:04.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:35:04.387 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.387 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.387 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.387 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:35:04.387 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:04.387 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.982630467000007, 2024-04-24T14:35:04.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:35:04.388 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.388 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:04.388 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:04.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:04.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:35:04.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:04.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:35:04.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:35:04.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:35:04.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:35:04.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:35:04.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:35:04.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:35:04.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:35:04.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:35:04.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:35:04.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:04.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:35:04.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:35:04.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:35:04.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:35:04.392 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.392 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.392 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.392 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:136 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'68, mlcod=62'66)", 2024-04-24T14:35:04.392 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:35:04.392 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.857694346000002, 2024-04-24T14:35:04.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039013530000000002, 2024-04-24T14:35:04.392 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.392 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:04.392 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:35:04.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501368+0000", 2024-04-24T14:35:04.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501366+0000", 2024-04-24T14:35:04.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:04.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501793+0000", 2024-04-24T14:35:04.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00042618499999999999 2024-04-24T14:35:04.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501823+0000", 2024-04-24T14:35:04.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0682999999999997e-05 2024-04-24T14:35:04.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501835+0000", 2024-04-24T14:35:04.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1837e-05 2024-04-24T14:35:04.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501947+0000", 2024-04-24T14:35:04.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000112163 2024-04-24T14:35:04.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.501991+0000", 2024-04-24T14:35:04.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.3908e-05 2024-04-24T14:35:04.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:04.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505159+0000", 2024-04-24T14:35:04.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0031674749999999999 2024-04-24T14:35:04.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.505270+0000", 2024-04-24T14:35:04.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000110902 2024-04-24T14:35:04.396 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.396 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.396 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.397 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:140 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'72, mlcod=62'70)", 2024-04-24T14:35:04.397 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:35:04.397 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.838306502999998, 2024-04-24T14:35:04.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043585669999999998, 2024-04-24T14:35:04.397 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.397 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:04.397 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.397 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:35:04.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520756+0000", 2024-04-24T14:35:04.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.520755+0000", 2024-04-24T14:35:04.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:04.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521286+0000", 2024-04-24T14:35:04.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053104900000000002 2024-04-24T14:35:04.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521300+0000", 2024-04-24T14:35:04.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4100999999999999e-05 2024-04-24T14:35:04.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521306+0000", 2024-04-24T14:35:04.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3369999999999996e-06 2024-04-24T14:35:04.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521363+0000", 2024-04-24T14:35:04.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6490999999999998e-05 2024-04-24T14:35:04.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.521399+0000", 2024-04-24T14:35:04.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6217999999999998e-05 2024-04-24T14:35:04.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:04.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525068+0000", 2024-04-24T14:35:04.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003669449 2024-04-24T14:35:04.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.525115+0000", 2024-04-24T14:35:04.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6161000000000001e-05 2024-04-24T14:35:04.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.401 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.402 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:35:04.402 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:04.402 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.828578660000005, 2024-04-24T14:35:04.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:35:04.402 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.402 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:04.402 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:04.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:04.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:35:04.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:04.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:35:04.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:35:04.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:35:04.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:35:04.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:35:04.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:35:04.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:35:04.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:35:04.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:35:04.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:35:04.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:04.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:35:04.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:35:04.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:35:04.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:35:04.406 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.406 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.406 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.407 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:174 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'78, mlcod=62'76)", 2024-04-24T14:35:04.407 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:35:04.407 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.66846898, 2024-04-24T14:35:04.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041852870000000002, 2024-04-24T14:35:04.407 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.407 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:04.407 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:35:04.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690594+0000", 2024-04-24T14:35:04.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.690592+0000", 2024-04-24T14:35:04.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:04.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691299+0000", 2024-04-24T14:35:04.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070759100000000004 2024-04-24T14:35:04.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691328+0000", 2024-04-24T14:35:04.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8986999999999999e-05 2024-04-24T14:35:04.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691345+0000", 2024-04-24T14:35:04.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6463000000000001e-05 2024-04-24T14:35:04.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691461+0000", 2024-04-24T14:35:04.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000116325 2024-04-24T14:35:04.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.691496+0000", 2024-04-24T14:35:04.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4474999999999998e-05 2024-04-24T14:35:04.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:04.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694702+0000", 2024-04-24T14:35:04.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003206478 2024-04-24T14:35:04.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.694779+0000", 2024-04-24T14:35:04.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6804000000000005e-05 2024-04-24T14:35:04.411 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.411 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.411 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.411 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:35:04.411 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:04.412 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.658739983999993, 2024-04-24T14:35:04.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:35:04.412 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.412 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:04.412 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:04.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:04.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:35:04.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:04.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:35:04.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:35:04.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:35:04.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:35:04.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:35:04.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:35:04.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:35:04.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:35:04.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:35:04.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:35:04.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:04.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:35:04.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:35:04.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:35:04.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:35:04.416 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.416 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.416 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.416 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:35:04.416 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:04.416 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.323769149, 2024-04-24T14:35:04.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:35:04.416 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.417 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:04.417 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:04.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:04.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:04.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:04.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:35:04.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:35:04.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:35:04.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:35:04.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:35:04.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:35:04.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:35:04.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:35:04.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:35:04.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:35:04.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:04.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:35:04.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:35:04.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:35:04.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:35:04.420 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.420 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.421 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.421 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:35:04.421 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:04.421 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 61.479618401000003, 2024-04-24T14:35:04.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:35:04.421 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.421 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:35:04.421 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.421 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.421 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:04.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:04.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:04.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:04.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:35:04.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:35:04.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:35:04.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:35:04.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:35:04.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:35:04.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:35:04.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:35:04.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:35:04.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:35:04.424 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.424 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.424 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.424 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4198.0:5 3.2 3.7c339972 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-24T14:35:04.425 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:35:04.425 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 51.151380758999998, 2024-04-24T14:35:04.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043717790000000001, 2024-04-24T14:35:04.425 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.425 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "queued for pg", 2024-04-24T14:35:04.425 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:04.425 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4198", 2024-04-24T14:35:04.425 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/1367638240", 2024-04-24T14:35:04.425 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 5 2024-04-24T14:35:04.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.425 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:35:04.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207682+0000", 2024-04-24T14:35:04.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207678+0000", 2024-04-24T14:35:04.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999962 2024-04-24T14:35:04.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207700+0000", 2024-04-24T14:35:04.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1313000000000001e-05 2024-04-24T14:35:04.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207757+0000", 2024-04-24T14:35:04.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7499e-05 2024-04-24T14:35:04.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.207779+0000", 2024-04-24T14:35:04.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1478999999999999e-05 2024-04-24T14:35:04.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:13.212054+0000", 2024-04-24T14:35:04.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042750849999999996 2024-04-24T14:35:04.428 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.428 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.428 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.428 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:35:04.428 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:04.429 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.255855400999998, 2024-04-24T14:35:04.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044084859999999997, 2024-04-24T14:35:04.429 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.429 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:04.429 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:04.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:04.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103206+0000", 2024-04-24T14:35:04.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:04.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104731+0000", 2024-04-24T14:35:04.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001524755 2024-04-24T14:35:04.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104759+0000", 2024-04-24T14:35:04.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7973000000000001e-05 2024-04-24T14:35:04.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104807+0000", 2024-04-24T14:35:04.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8112000000000003e-05 2024-04-24T14:35:04.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104907+0000", 2024-04-24T14:35:04.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100317 2024-04-24T14:35:04.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104918+0000", 2024-04-24T14:35:04.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747999999999999e-05 2024-04-24T14:35:04.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.107616+0000", 2024-04-24T14:35:04.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026977559999999999 2024-04-24T14:35:04.432 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.432 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.432 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.433 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(3.2 83/82 [PushOp(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 62'74, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@62'74, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:35:04.433 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:35:04.433 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.249461023999999, 2024-04-24T14:35:04.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041425760000000002, 2024-04-24T14:35:04.433 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.433 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:04.433 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:35:04.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109602+0000", 2024-04-24T14:35:04.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.109600+0000", 2024-04-24T14:35:04.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:04.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111210+0000", 2024-04-24T14:35:04.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016097279999999999 2024-04-24T14:35:04.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111237+0000", 2024-04-24T14:35:04.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7189999999999999e-05 2024-04-24T14:35:04.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111251+0000", 2024-04-24T14:35:04.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.365e-05 2024-04-24T14:35:04.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111367+0000", 2024-04-24T14:35:04.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000115703 2024-04-24T14:35:04.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.111378+0000", 2024-04-24T14:35:04.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1873e-05 2024-04-24T14:35:04.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.113744+0000", 2024-04-24T14:35:04.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0023657069999999999 2024-04-24T14:35:04.436 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.436 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.437 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.437 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:35:04.437 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:04.437 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.232560448000001, 2024-04-24T14:35:04.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046193859999999996, 2024-04-24T14:35:04.437 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.437 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:04.437 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:04.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:04.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126501+0000", 2024-04-24T14:35:04.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:04.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128169+0000", 2024-04-24T14:35:04.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016686229999999999 2024-04-24T14:35:04.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128202+0000", 2024-04-24T14:35:04.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2539000000000001e-05 2024-04-24T14:35:04.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128222+0000", 2024-04-24T14:35:04.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0149e-05 2024-04-24T14:35:04.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128375+0000", 2024-04-24T14:35:04.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000153459 2024-04-24T14:35:04.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128399+0000", 2024-04-24T14:35:04.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3393000000000001e-05 2024-04-24T14:35:04.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.131122+0000", 2024-04-24T14:35:04.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027227140000000002 2024-04-24T14:35:04.441 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.441 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.441 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.441 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 49'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@49'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:35:04.441 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:35:04.441 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 46.210779068000001, 2024-04-24T14:35:04.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0038414529999999999, 2024-04-24T14:35:04.441 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:04.442 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:04.442 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:04.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:04.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:35:04.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:04.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148284+0000", 2024-04-24T14:35:04.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:04.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.148282+0000", 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149875+0000", 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0015929900000000001 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149908+0000", 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3102000000000003e-05 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:04.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.149941+0000", 2024-04-24T14:35:04.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2480000000000001e-05 2024-04-24T14:35:04.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:04.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150031+0000", 2024-04-24T14:35:04.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0394000000000002e-05 2024-04-24T14:35:04.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:04.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.150043+0000", 2024-04-24T14:35:04.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1967000000000001e-05 2024-04-24T14:35:04.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:04.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:04.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:04.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.152125+0000", 2024-04-24T14:35:04.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020817029999999999 2024-04-24T14:35:04.445 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.445 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.445 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.445 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:04.445 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:04.445 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:04.447 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:35:04.447 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:35:04.448 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:35:04.448 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:04.448+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:04.451 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo blkdiscard --offset 0 --length 4608 /dev/nbd1 2024-04-24T14:35:04.451 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:35:04.451 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:35:04.451 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:35:04.451 INFO:tasks.workunit.client.0.smithi173.stderr:++ which blkdiscard 2024-04-24T14:35:04.452 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T14:35:04.452 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:35:04.452 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4608 /dev/nbd1 2024-04-24T14:35:04.452 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo blkdiscard --offset 512 --length 7168 /dev/nbd1 2024-04-24T14:35:04.452 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:35:04.452 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:35:04.452 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:35:04.452 INFO:tasks.workunit.client.0.smithi173.stderr:++ which blkdiscard 2024-04-24T14:35:04.452 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T14:35:04.452 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:35:04.452 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7168 /dev/nbd1 2024-04-24T14:35:04.452 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo blkdiscard --offset 512 --length 7680 /dev/nbd1 2024-04-24T14:35:04.452 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:35:04.452 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:35:04.453 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:35:04.453 INFO:tasks.workunit.client.0.smithi173.stderr:++ which blkdiscard 2024-04-24T14:35:04.453 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T14:35:04.453 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:35:04.453 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7680 /dev/nbd1 2024-04-24T14:35:04.453 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo blkdiscard --offset 512 --length 8192 /dev/nbd1 2024-04-24T14:35:04.453 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:35:04.453 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:35:04.453 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:35:04.453 INFO:tasks.workunit.client.0.smithi173.stderr:++ which blkdiscard 2024-04-24T14:35:04.453 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T14:35:04.453 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:35:04.453 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 8192 /dev/nbd1 2024-04-24T14:35:04.453 INFO:tasks.workunit.client.0.smithi173.stderr:+ tries=0 2024-04-24T14:35:04.454 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd16809 2024-04-24T14:35:04.454 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '/entries inspected/ {print $1}' 2024-04-24T14:35:04.454 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:35:04.473 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:35:04.499 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:35:04.526 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:35:04.528 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:35:04.528 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:35:04.528 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:35:04.528 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:35:04.528 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:35:04.528 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:35:04.544 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:35:04.548+0000 7f2fa67fc700 -1 JournalMetadata: 0x55b3f75b1230 failed to locate client: INSPECT 2024-04-24T14:35:04.547 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:04.552+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:04.549 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:04.549 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:04.554 INFO:tasks.workunit.client.0.smithi173.stderr:+ queue_length=4 2024-04-24T14:35:04.554 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 0 -lt 10 ']' 2024-04-24T14:35:04.554 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 4 -gt 0 ']' 2024-04-24T14:35:04.554 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd16809 --verbose 2024-04-24T14:35:04.558 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:35:04.644 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=1 2024-04-24T14:35:04.644 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:04.644 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:04.644 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 0, 2024-04-24T14:35:04.645 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:04.645 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:04.645 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.362976+0000" 2024-04-24T14:35:04.645 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:04.645 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=2 2024-04-24T14:35:04.645 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:04.645 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:04.645 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 0, 2024-04-24T14:35:04.645 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:04.645 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:04.645 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.383412+0000" 2024-04-24T14:35:04.645 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:04.646 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=3 2024-04-24T14:35:04.646 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:04.646 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:04.646 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 4096, 2024-04-24T14:35:04.646 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:04.646 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:04.646 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.416941+0000" 2024-04-24T14:35:04.646 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:04.646 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=4 2024-04-24T14:35:04.647 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:04.647 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:04.647 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 4096, 2024-04-24T14:35:04.647 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:04.647 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:04.647 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.439515+0000" 2024-04-24T14:35:04.647 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:04.647 INFO:tasks.workunit.client.0.smithi173.stdout:Summary: 2024-04-24T14:35:04.647 INFO:tasks.workunit.client.0.smithi173.stdout: 4 entries inspected, 0 errors 2024-04-24T14:35:04.648 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:04.652+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:04.657 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:35:04.660+0000 7f83211c3700 -1 JournalMetadata: 0x5620869c1430 failed to locate client: INSPECT 2024-04-24T14:35:04.663 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 1 2024-04-24T14:35:04.702 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:04.702 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:04.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:35:04.749 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:04.752+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:04.814 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:04.814 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:04.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:35:04.850 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:04.854+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:04.923 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:04.924 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:04.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:35:04.978 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:04.982+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:05.034 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:05.034 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:05.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:35:05.052 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:05.056+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:05.138 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:05.138 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:05.147 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:35:05.152 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:05.156+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:05.249 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:05.249 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:05.253 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:05.256+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:05.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:35:05.310 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-24T14:35:05.353 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:05.354+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:05.363 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:05.363 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:05.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:35:05.454 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:05.458+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:05.465 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:05.465 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:05.474 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:35:05.555 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:05.560+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:05.582 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:05.582 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:05.592 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:35:05.655 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:05.660+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:05.666 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd16809 2024-04-24T14:35:05.666 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '/entries inspected/ {print $1}' 2024-04-24T14:35:05.687 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:05.687 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:05.695 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:35:05.756 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:05.760+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:05.765 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:35:05.768+0000 7f4871ffb700 -1 JournalMetadata: 0x557d32ccb8c0 failed to locate client: INSPECT 2024-04-24T14:35:05.772 INFO:tasks.workunit.client.0.smithi173.stderr:+ queue_length=4 2024-04-24T14:35:05.772 INFO:tasks.workunit.client.0.smithi173.stderr:+ tries=1 2024-04-24T14:35:05.773 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1 -lt 10 ']' 2024-04-24T14:35:05.773 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 4 -gt 0 ']' 2024-04-24T14:35:05.773 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd16809 --verbose 2024-04-24T14:35:05.797 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:05.797 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:05.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:35:05.857 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:05.858+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:05.857 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=1 2024-04-24T14:35:05.858 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:05.858 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:05.858 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 0, 2024-04-24T14:35:05.858 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:05.858 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:05.858 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.362976+0000" 2024-04-24T14:35:05.858 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:05.858 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=2 2024-04-24T14:35:05.858 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:05.858 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:05.858 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 0, 2024-04-24T14:35:05.858 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:05.859 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:05.859 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.383412+0000" 2024-04-24T14:35:05.859 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:05.859 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=3 2024-04-24T14:35:05.859 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:05.859 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:05.859 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 4096, 2024-04-24T14:35:05.859 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:05.859 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:05.859 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.416941+0000" 2024-04-24T14:35:05.859 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:05.859 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=4 2024-04-24T14:35:05.859 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:05.859 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:05.860 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 4096, 2024-04-24T14:35:05.860 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:05.860 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:05.860 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.439515+0000" 2024-04-24T14:35:05.860 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:05.860 INFO:tasks.workunit.client.0.smithi173.stdout:Summary: 2024-04-24T14:35:05.860 INFO:tasks.workunit.client.0.smithi173.stdout: 4 entries inspected, 0 errors 2024-04-24T14:35:05.867 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:35:05.872+0000 7efc06ffd700 -1 JournalMetadata: 0x564a10febec0 failed to locate client: INSPECT 2024-04-24T14:35:05.873 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 1 2024-04-24T14:35:05.949 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:05.950 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:05.958 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:05.960+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:05.959 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:35:06.055 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:06.055 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:06.058 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:06.064+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:06.064 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:35:06.159 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:06.164+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:06.164 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:06.164 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:06.173 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:35:06.259 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:06.264+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:06.271 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:06.271 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:06.280 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:35:06.361 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:06.364+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:06.374 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:06.374 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:06.383 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:35:06.462 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:06.466+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:06.528 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:06.528 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:06.537 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:35:06.543 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:35:06.582 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:06.586+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:06.663 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:06.663 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:06.663 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:06.666+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:06.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:35:06.724 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:35:06.744 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:35:06.762 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:35:06.764 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:06.768+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:06.783 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:35:06.784 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:06.784 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:06.784 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:06.784 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:06.784 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:06.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:35:06.803 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:35:06.825 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:35:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:35:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:35:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:35:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:35:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:35:06.827 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:35:06.865 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:06.866+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:06.876 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd16809 2024-04-24T14:35:06.876 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '/entries inspected/ {print $1}' 2024-04-24T14:35:06.892 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:06.892 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:06.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:06.892 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:06.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.892 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:06.892 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.015687710999998, 2024-04-24T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.899 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.899 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:06.899 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:06.899 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.015658725999998, 2024-04-24T14:35:06.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:35:06.899 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.899 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:06.899 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.899 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:06.899 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:06.900 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:35:06.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.900 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:06.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:06.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:06.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:06.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:35:06.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:35:06.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:35:06.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:35:06.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:35:06.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:35:06.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:35:06.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:35:06.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:06.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:35:06.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:35:06.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:35:06.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:35:06.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:06.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:35:06.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:35:06.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:35:06.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:35:06.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:35:06.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:35:06.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:35:06.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:35:06.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.905 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.905 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:06.905 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:06.905 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.015640241, 2024-04-24T14:35:06.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:35:06.905 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.905 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:06.905 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.906 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:06.906 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:06.906 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:35:06.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.906 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:06.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:06.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:35:06.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:06.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:35:06.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:35:06.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:35:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:35:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:35:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:35:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:35:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:35:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:35:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:35:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:35:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:35:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:35:06.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:35:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:35:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.015624959999997, 2024-04-24T14:35:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:35:06.911 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:35:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:35:06.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:35:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:35:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:35:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:35:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:35:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:35:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:35:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:35:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:35:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:35:06.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:35:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:35:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:35:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:35:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:35:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:35:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:35:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:35:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:35:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.917 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.015609178999995, 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:06.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:06.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:06.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:06.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:35:06.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:35:06.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:35:06.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:35:06.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:35:06.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:35:06.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:06.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:35:06.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:35:06.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:35:06.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:35:06.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:35:06.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:35:06.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:35:06.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:35:06.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.923 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.923 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:06.923 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.015594891000006, 2024-04-24T14:35:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:35:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:35:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:35:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:35:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:35:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:35:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:35:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:35:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:35:06.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:35:06.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:06.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:35:06.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:35:06.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:35:06.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:35:06.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:06.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:35:06.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:35:06.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:35:06.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:35:06.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:35:06.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:35:06.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:35:06.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:35:06.929 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.929 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.929 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.930 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:06.930 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:06.930 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.015575139000006, 2024-04-24T14:35:06.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:35:06.930 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.930 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:06.930 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.930 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:06.930 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:06.930 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:35:06.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.931 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:06.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:06.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:06.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:06.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:35:06.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:35:06.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:35:06.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:35:06.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:35:06.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:35:06.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:35:06.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:35:06.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:06.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:35:06.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:35:06.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:35:06.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:35:06.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:06.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:35:06.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:35:06.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:35:06.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:35:06.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:35:06.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:35:06.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:35:06.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:35:06.936 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.936 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.936 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.936 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:06.936 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:06.936 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.015557692000002, 2024-04-24T14:35:06.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:35:06.936 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.936 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:06.936 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.936 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:06.937 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:06.937 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:35:06.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.937 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:06.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:06.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:06.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:06.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:35:06.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:35:06.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:35:06.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:35:06.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:35:06.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:35:06.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:35:06.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:35:06.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:06.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:35:06.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:35:06.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:35:06.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:35:06.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:06.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:35:06.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:35:06.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:35:06.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:35:06.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:35:06.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:35:06.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:35:06.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:35:06.942 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.942 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.942 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.942 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:100 3.2 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:06.942 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:06.942 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.141502877000001, 2024-04-24T14:35:06.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009078015, 2024-04-24T14:35:06.942 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.943 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:06.943 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.943 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:06.943 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:06.943 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 100 2024-04-24T14:35:06.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.943 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:06.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:06.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754019+0000", 2024-04-24T14:35:06.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:06.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754025+0000", 2024-04-24T14:35:06.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2110000000000003e-06 2024-04-24T14:35:06.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754037+0000", 2024-04-24T14:35:06.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1868999999999999e-05 2024-04-24T14:35:06.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754046+0000", 2024-04-24T14:35:06.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7430000000000008e-06 2024-04-24T14:35:06.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754095+0000", 2024-04-24T14:35:06.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8924000000000001e-05 2024-04-24T14:35:06.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754153+0000", 2024-04-24T14:35:06.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8165000000000003e-05 2024-04-24T14:35:06.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:06.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754262+0000", 2024-04-24T14:35:06.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108694 2024-04-24T14:35:06.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:06.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.762999+0000", 2024-04-24T14:35:06.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087372590000000007 2024-04-24T14:35:06.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:06.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763027+0000", 2024-04-24T14:35:06.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8467e-05 2024-04-24T14:35:06.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:06.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763070+0000", 2024-04-24T14:35:06.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2236e-05 2024-04-24T14:35:06.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763098+0000", 2024-04-24T14:35:06.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8561999999999999e-05 2024-04-24T14:35:06.948 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.948 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.948 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.948 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:101 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [delete] snapc 7=[7,6,4] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:06.948 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:06.948 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.936295211000001, 2024-04-24T14:35:06.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012894743, 2024-04-24T14:35:06.949 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.949 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:06.949 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.949 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:06.949 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:06.949 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 101 2024-04-24T14:35:06.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.949 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:06.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:06.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959227+0000", 2024-04-24T14:35:06.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:06.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959233+0000", 2024-04-24T14:35:06.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.857e-06 2024-04-24T14:35:06.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959249+0000", 2024-04-24T14:35:06.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6155999999999999e-05 2024-04-24T14:35:06.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959257+0000", 2024-04-24T14:35:06.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7440000000000007e-06 2024-04-24T14:35:06.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959790+0000", 2024-04-24T14:35:06.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053381100000000003 2024-04-24T14:35:06.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960206+0000", 2024-04-24T14:35:06.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041549700000000002 2024-04-24T14:35:06.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:06.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960744+0000", 2024-04-24T14:35:06.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053762199999999997 2024-04-24T14:35:06.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:06.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971963+0000", 2024-04-24T14:35:06.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011219816000000001 2024-04-24T14:35:06.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:06.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972021+0000", 2024-04-24T14:35:06.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7855000000000001e-05 2024-04-24T14:35:06.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:06.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972073+0000", 2024-04-24T14:35:06.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2196e-05 2024-04-24T14:35:06.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972123+0000", 2024-04-24T14:35:06.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9311999999999998e-05 2024-04-24T14:35:06.954 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.954 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.954 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.954 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:102 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:06.954 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.936255185, 2024-04-24T14:35:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012939713, 2024-04-24T14:35:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 102 2024-04-24T14:35:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959267+0000", 2024-04-24T14:35:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959274+0000", 2024-04-24T14:35:06.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.593e-06 2024-04-24T14:35:06.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959291+0000", 2024-04-24T14:35:06.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6387000000000001e-05 2024-04-24T14:35:06.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959301+0000", 2024-04-24T14:35:06.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0023e-05 2024-04-24T14:35:06.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969594+0000", 2024-04-24T14:35:06.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010293182 2024-04-24T14:35:06.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969902+0000", 2024-04-24T14:35:06.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030835299999999998 2024-04-24T14:35:06.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:06.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970408+0000", 2024-04-24T14:35:06.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050588100000000002 2024-04-24T14:35:06.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:06.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972009+0000", 2024-04-24T14:35:06.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016011210000000001 2024-04-24T14:35:06.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:06.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972132+0000", 2024-04-24T14:35:06.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122521 2024-04-24T14:35:06.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:06.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972174+0000", 2024-04-24T14:35:06.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2242e-05 2024-04-24T14:35:06.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972208+0000", 2024-04-24T14:35:06.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3522999999999997e-05 2024-04-24T14:35:06.960 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.960 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.960 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.960 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:104 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:06.960 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:06.961 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.935643970999998, 2024-04-24T14:35:06.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014589062, 2024-04-24T14:35:06.961 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.961 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:06.961 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.961 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:06.961 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:06.961 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 104 2024-04-24T14:35:06.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.961 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959877+0000", 2024-04-24T14:35:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:06.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959890+0000", 2024-04-24T14:35:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2537e-05 2024-04-24T14:35:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959909+0000", 2024-04-24T14:35:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9340000000000001e-05 2024-04-24T14:35:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959922+0000", 2024-04-24T14:35:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2882e-05 2024-04-24T14:35:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970900+0000", 2024-04-24T14:35:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010977736 2024-04-24T14:35:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971039+0000", 2024-04-24T14:35:06.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139593 2024-04-24T14:35:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971453+0000", 2024-04-24T14:35:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041351099999999998 2024-04-24T14:35:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974044+0000", 2024-04-24T14:35:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025908289999999998 2024-04-24T14:35:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974377+0000", 2024-04-24T14:35:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033310399999999997 2024-04-24T14:35:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974433+0000", 2024-04-24T14:35:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6166999999999999e-05 2024-04-24T14:35:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974468+0000", 2024-04-24T14:35:06.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5336000000000002e-05 2024-04-24T14:35:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:106 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.935584638000002, 2024-04-24T14:35:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017192064999999999, 2024-04-24T14:35:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 106 2024-04-24T14:35:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959938+0000", 2024-04-24T14:35:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959943+0000", 2024-04-24T14:35:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.677e-06 2024-04-24T14:35:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959953+0000", 2024-04-24T14:35:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8069999999999994e-06 2024-04-24T14:35:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:35:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0560000000000001e-06 2024-04-24T14:35:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972232+0000", 2024-04-24T14:35:06.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012272460000000001 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972304+0000", 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2630000000000001e-05 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972548+0000", 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000244047 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976900+0000", 2024-04-24T14:35:06.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043514069999999998 2024-04-24T14:35:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977076+0000", 2024-04-24T14:35:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176281 2024-04-24T14:35:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977106+0000", 2024-04-24T14:35:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0015e-05 2024-04-24T14:35:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977131+0000", 2024-04-24T14:35:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4669e-05 2024-04-24T14:35:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:114 3.2 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.912263474, 2024-04-24T14:35:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090880089999999993, 2024-04-24T14:35:06.973 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 114 2024-04-24T14:35:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983259+0000", 2024-04-24T14:35:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983264+0000", 2024-04-24T14:35:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9130000000000002e-06 2024-04-24T14:35:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983277+0000", 2024-04-24T14:35:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3723999999999999e-05 2024-04-24T14:35:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983282+0000", 2024-04-24T14:35:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0429999999999998e-06 2024-04-24T14:35:06.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983339+0000", 2024-04-24T14:35:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6323000000000003e-05 2024-04-24T14:35:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983464+0000", 2024-04-24T14:35:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012554299999999999 2024-04-24T14:35:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:06.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983693+0000", 2024-04-24T14:35:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000228848 2024-04-24T14:35:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.986978+0000", 2024-04-24T14:35:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032852810000000001 2024-04-24T14:35:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992288+0000", 2024-04-24T14:35:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053095369999999996 2024-04-24T14:35:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992322+0000", 2024-04-24T14:35:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3683999999999998e-05 2024-04-24T14:35:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992348+0000", 2024-04-24T14:35:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6157999999999999e-05 2024-04-24T14:35:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:121 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.900207375000001, 2024-04-24T14:35:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098798159999999996, 2024-04-24T14:35:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 121 2024-04-24T14:35:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995315+0000", 2024-04-24T14:35:06.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995320+0000", 2024-04-24T14:35:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0239999999999997e-06 2024-04-24T14:35:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995328+0000", 2024-04-24T14:35:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7800000000000001e-06 2024-04-24T14:35:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995334+0000", 2024-04-24T14:35:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2330000000000003e-06 2024-04-24T14:35:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998810+0000", 2024-04-24T14:35:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034756090000000002 2024-04-24T14:35:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998870+0000", 2024-04-24T14:35:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0625999999999999e-05 2024-04-24T14:35:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.999091+0000", 2024-04-24T14:35:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000220631 2024-04-24T14:35:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.001389+0000", 2024-04-24T14:35:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022977549999999998 2024-04-24T14:35:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:06.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005097+0000", 2024-04-24T14:35:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037085439999999998 2024-04-24T14:35:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005154+0000", 2024-04-24T14:35:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6208000000000001e-05 2024-04-24T14:35:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005196+0000", 2024-04-24T14:35:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2052000000000003e-05 2024-04-24T14:35:06.985 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:127 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.889905859000002, 2024-04-24T14:35:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013414918, 2024-04-24T14:35:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2024-04-24T14:35:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005616+0000", 2024-04-24T14:35:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005622+0000", 2024-04-24T14:35:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2020000000000003e-06 2024-04-24T14:35:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005632+0000", 2024-04-24T14:35:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2024-04-24T14:35:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005637+0000", 2024-04-24T14:35:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7439999999999998e-06 2024-04-24T14:35:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005947+0000", 2024-04-24T14:35:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000310792 2024-04-24T14:35:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006059+0000", 2024-04-24T14:35:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011192399999999999 2024-04-24T14:35:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006503+0000", 2024-04-24T14:35:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000443394 2024-04-24T14:35:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018864+0000", 2024-04-24T14:35:06.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012361162 2024-04-24T14:35:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018877+0000", 2024-04-24T14:35:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3497e-05 2024-04-24T14:35:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018965+0000", 2024-04-24T14:35:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7476000000000001e-05 2024-04-24T14:35:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019032+0000", 2024-04-24T14:35:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7469000000000001e-05 2024-04-24T14:35:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:129 3.2 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.889765625999999, 2024-04-24T14:35:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010059754000000001, 2024-04-24T14:35:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.992 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 129 2024-04-24T14:35:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:06.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005757+0000", 2024-04-24T14:35:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005760+0000", 2024-04-24T14:35:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2430000000000001e-06 2024-04-24T14:35:06.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005769+0000", 2024-04-24T14:35:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4810000000000006e-06 2024-04-24T14:35:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005773+0000", 2024-04-24T14:35:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:35:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005840+0000", 2024-04-24T14:35:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6934999999999995e-05 2024-04-24T14:35:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005961+0000", 2024-04-24T14:35:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012153 2024-04-24T14:35:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006181+0000", 2024-04-24T14:35:06.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021947500000000001 2024-04-24T14:35:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015520+0000", 2024-04-24T14:35:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093393049999999991 2024-04-24T14:35:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015733+0000", 2024-04-24T14:35:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021355699999999999 2024-04-24T14:35:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015773+0000", 2024-04-24T14:35:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9554000000000001e-05 2024-04-24T14:35:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015817+0000", 2024-04-24T14:35:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4325e-05 2024-04-24T14:35:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:06.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:131 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.889487744999997, 2024-04-24T14:35:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014251976, 2024-04-24T14:35:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:06.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131 2024-04-24T14:35:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006034+0000", 2024-04-24T14:35:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006042+0000", 2024-04-24T14:35:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2970000000000004e-06 2024-04-24T14:35:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006061+0000", 2024-04-24T14:35:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8505999999999999e-05 2024-04-24T14:35:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006071+0000", 2024-04-24T14:35:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0409e-05 2024-04-24T14:35:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007899+0000", 2024-04-24T14:35:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018274789999999999 2024-04-24T14:35:07.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008281+0000", 2024-04-24T14:35:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038244700000000001 2024-04-24T14:35:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008528+0000", 2024-04-24T14:35:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024723900000000001 2024-04-24T14:35:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:07.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019070+0000", 2024-04-24T14:35:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010541867999999999 2024-04-24T14:35:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020099+0000", 2024-04-24T14:35:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00102852 2024-04-24T14:35:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020226+0000", 2024-04-24T14:35:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012744300000000001 2024-04-24T14:35:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020287+0000", 2024-04-24T14:35:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1253000000000005e-05 2024-04-24T14:35:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:132 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.889433892, 2024-04-24T14:35:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015917981000000001, 2024-04-24T14:35:07.005 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 132 2024-04-24T14:35:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:07.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006093+0000", 2024-04-24T14:35:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3819999999999997e-06 2024-04-24T14:35:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006103+0000", 2024-04-24T14:35:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0352e-05 2024-04-24T14:35:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006110+0000", 2024-04-24T14:35:07.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3910000000000003e-06 2024-04-24T14:35:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015586+0000", 2024-04-24T14:35:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094759119999999995 2024-04-24T14:35:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015730+0000", 2024-04-24T14:35:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014487199999999999 2024-04-24T14:35:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015886+0000", 2024-04-24T14:35:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015573799999999999 2024-04-24T14:35:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020306+0000", 2024-04-24T14:35:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044199399999999998 2024-04-24T14:35:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021887+0000", 2024-04-24T14:35:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015809400000000001 2024-04-24T14:35:07.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021942+0000", 2024-04-24T14:35:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5071000000000002e-05 2024-04-24T14:35:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.022007+0000", 2024-04-24T14:35:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5110000000000005e-05 2024-04-24T14:35:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.5324.0:127 3.0 e87/69 ondisk, result = 0)", 2024-04-24T14:35:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.887724824000003, 2024-04-24T14:35:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011245125, 2024-04-24T14:35:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007804+0000", 2024-04-24T14:35:07.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6610000000000003e-06 2024-04-24T14:35:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007817+0000", 2024-04-24T14:35:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3030999999999999e-05 2024-04-24T14:35:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007832+0000", 2024-04-24T14:35:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.47e-05 2024-04-24T14:35:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018843+0000", 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011010647 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018853+0000", 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319000000000001e-05 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019043+0000", 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019055600000000001 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.015 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.016 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.016 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:07.018 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:35:06.980+0000 7fb5a908f700 -1 JournalMetadata: 0x563691a62dc0 failed to locate client: INSPECT 2024-04-24T14:35:07.018 INFO:tasks.workunit.client.0.smithi173.stderr:+ queue_length=4 2024-04-24T14:35:07.018 INFO:tasks.workunit.client.0.smithi173.stderr:+ tries=2 2024-04-24T14:35:07.018 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 2 -lt 10 ']' 2024-04-24T14:35:07.018 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 4 -gt 0 ']' 2024-04-24T14:35:07.018 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd16809 --verbose 2024-04-24T14:35:07.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:35:07.019 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:07.020+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:07.078 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=1 2024-04-24T14:35:07.078 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:07.078 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:07.078 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 0, 2024-04-24T14:35:07.078 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:07.078 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:07.078 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.362976+0000" 2024-04-24T14:35:07.078 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:07.078 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=2 2024-04-24T14:35:07.079 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:07.079 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:07.079 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 0, 2024-04-24T14:35:07.079 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:07.079 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:07.079 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.383412+0000" 2024-04-24T14:35:07.079 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:07.079 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=3 2024-04-24T14:35:07.079 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:07.079 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:07.079 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 4096, 2024-04-24T14:35:07.079 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:07.079 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:07.079 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.416941+0000" 2024-04-24T14:35:07.080 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:07.080 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=4 2024-04-24T14:35:07.080 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:07.080 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:07.080 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 4096, 2024-04-24T14:35:07.080 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:07.080 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:07.080 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.439515+0000" 2024-04-24T14:35:07.080 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:07.080 INFO:tasks.workunit.client.0.smithi173.stdout:Summary: 2024-04-24T14:35:07.081 INFO:tasks.workunit.client.0.smithi173.stdout: 4 entries inspected, 0 errors 2024-04-24T14:35:07.086 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:35:07.092+0000 7fc5a3fff700 -1 JournalMetadata: 0x55c25c615b40 failed to locate client: INSPECT 2024-04-24T14:35:07.092 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 1 2024-04-24T14:35:07.117 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:07.122+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:07.129 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:07.129 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:07.129 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:35:07.129 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:07.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:35:07.219 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:07.222+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:07.241 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:07.241 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:07.241 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:07.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:35:07.319 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:07.322+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:07.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T14:35:07.372 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:07.372 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:07.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:07.372 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.619079736000003, 2024-04-24T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:35:07.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:35:07.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:35:07.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:35:07.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:35:07.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:35:07.379 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.379 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.379 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.379 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.379 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:07.379 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.619028901999997, 2024-04-24T14:35:07.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:35:07.379 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.379 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.380 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:35:07.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.380 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:07.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:07.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:07.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:07.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:35:07.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:35:07.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:35:07.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:35:07.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:35:07.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:35:07.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:35:07.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:35:07.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:35:07.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:35:07.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:35:07.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:35:07.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:35:07.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:35:07.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:35:07.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:35:07.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:35:07.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:35:07.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:35:07.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:35:07.385 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.385 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.385 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.385 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.385 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:07.385 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.618992594999995, 2024-04-24T14:35:07.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:35:07.385 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.386 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.386 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.386 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.386 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.386 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:35:07.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.386 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:07.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:07.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:35:07.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:07.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:35:07.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:35:07.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:35:07.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:35:07.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:35:07.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:35:07.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:35:07.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:35:07.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:35:07.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:35:07.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:35:07.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:35:07.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:35:07.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:35:07.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:35:07.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:35:07.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:35:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:35:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:35:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:35:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.392 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.392 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:07.392 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.615581344999995, 2024-04-24T14:35:07.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:35:07.392 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.392 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.392 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.392 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.392 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.392 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:35:07.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.392 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:07.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:07.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:35:07.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:07.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.615519668000005, 2024-04-24T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.615485092, 2024-04-24T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.615455753000006, 2024-04-24T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:07.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:07.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:35:07.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:35:07.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:35:07.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:35:07.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:35:07.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:35:07.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:35:07.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:35:07.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:35:07.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:35:07.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:35:07.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:35:07.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:35:07.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:35:07.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:35:07.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:35:07.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:35:07.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:35:07.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:35:07.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:35:07.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:35:07.417 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.417 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.417 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.417 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.417 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:07.417 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.615379171000001, 2024-04-24T14:35:07.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:35:07.417 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.417 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.418 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.418 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.418 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:35:07.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.418 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:07.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:07.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:35:07.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:35:07.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:35:07.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:35:07.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:35:07.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:35:07.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:35:07.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:35:07.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:35:07.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:35:07.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:35:07.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:35:07.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:35:07.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:35:07.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:35:07.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:35:07.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:35:07.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:35:07.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:35:07.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:35:07.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:35:07.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.423 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.424 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.424 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.424 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:07.424 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.615331626, 2024-04-24T14:35:07.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:35:07.424 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.424 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.424 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:35:07.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.425 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:07.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:07.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:35:07.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:35:07.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:35:07.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:35:07.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:35:07.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:35:07.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:35:07.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:35:07.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:35:07.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:35:07.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:35:07.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:35:07.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:35:07.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:35:07.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:35:07.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:35:07.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:35:07.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:35:07.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:35:07.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:35:07.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:35:07.430 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.430 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.430 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.430 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.430 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:07.431 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.615290983999998, 2024-04-24T14:35:07.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:35:07.431 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.431 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.431 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.431 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.431 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.431 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:35:07.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.431 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:07.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:07.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:07.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:07.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:35:07.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:35:07.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:35:07.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:35:07.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:35:07.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:35:07.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:35:07.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:35:07.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:35:07.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:35:07.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:35:07.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:35:07.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:35:07.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:35:07.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:35:07.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:35:07.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:35:07.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:35:07.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:35:07.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:35:07.437 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.437 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.437 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.437 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.437 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:07.437 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.615273873000007, 2024-04-24T14:35:07.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:35:07.437 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.438 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.438 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.438 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.438 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.438 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:35:07.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.438 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:07.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:07.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:35:07.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:07.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:35:07.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:35:07.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:35:07.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:35:07.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:35:07.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:35:07.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:35:07.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:35:07.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:35:07.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:35:07.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:35:07.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:35:07.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:35:07.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:35:07.443 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.443 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.443 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.443 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:103 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:07.443 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:07.443 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.415707759999997, 2024-04-24T14:35:07.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032646351999999997, 2024-04-24T14:35:07.444 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.444 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:07.444 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.444 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:07.444 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:07.444 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 103 2024-04-24T14:35:07.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.444 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:07.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:07.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959674+0000", 2024-04-24T14:35:07.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:07.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959688+0000", 2024-04-24T14:35:07.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2024-04-24T14:35:07.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959722+0000", 2024-04-24T14:35:07.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3936000000000002e-05 2024-04-24T14:35:07.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959735+0000", 2024-04-24T14:35:07.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2926e-05 2024-04-24T14:35:07.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959921+0000", 2024-04-24T14:35:07.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185601 2024-04-24T14:35:07.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960324+0000", 2024-04-24T14:35:07.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040356700000000002 2024-04-24T14:35:07.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:07.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960839+0000", 2024-04-24T14:35:07.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051410599999999996 2024-04-24T14:35:07.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:07.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.975634+0000", 2024-04-24T14:35:07.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014795770999999999 2024-04-24T14:35:07.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:07.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992113+0000", 2024-04-24T14:35:07.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016478982999999999 2024-04-24T14:35:07.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:07.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992228+0000", 2024-04-24T14:35:07.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114295 2024-04-24T14:35:07.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992323+0000", 2024-04-24T14:35:07.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5344000000000001e-05 2024-04-24T14:35:07.449 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.449 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.450 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.450 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:105 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:07.450 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:07.450 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.415457982, 2024-04-24T14:35:07.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032581757000000003, 2024-04-24T14:35:07.450 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.450 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:07.450 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.450 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:07.451 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:07.451 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105 2024-04-24T14:35:07.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.451 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:07.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:07.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959925+0000", 2024-04-24T14:35:07.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:07.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959937+0000", 2024-04-24T14:35:07.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2391999999999999e-05 2024-04-24T14:35:07.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:35:07.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2535999999999999e-05 2024-04-24T14:35:07.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959970+0000", 2024-04-24T14:35:07.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1039999999999999e-05 2024-04-24T14:35:07.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969531+0000", 2024-04-24T14:35:07.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095603949999999993 2024-04-24T14:35:07.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969669+0000", 2024-04-24T14:35:07.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013806499999999999 2024-04-24T14:35:07.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:07.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970010+0000", 2024-04-24T14:35:07.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034062200000000001 2024-04-24T14:35:07.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:07.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976983+0000", 2024-04-24T14:35:07.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069735600000000002 2024-04-24T14:35:07.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:07.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992413+0000", 2024-04-24T14:35:07.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015429841999999999 2024-04-24T14:35:07.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:07.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992457+0000", 2024-04-24T14:35:07.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4169999999999999e-05 2024-04-24T14:35:07.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992508+0000", 2024-04-24T14:35:07.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0995000000000002e-05 2024-04-24T14:35:07.456 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.456 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.456 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.456 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:113 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:07.457 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:07.457 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.395231690999999, 2024-04-24T14:35:07.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014793863000000001, 2024-04-24T14:35:07.457 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.457 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:07.457 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.457 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:07.457 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:07.457 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 113 2024-04-24T14:35:07.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.457 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:07.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:07.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980152+0000", 2024-04-24T14:35:07.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:07.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980156+0000", 2024-04-24T14:35:07.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.619e-06 2024-04-24T14:35:07.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980163+0000", 2024-04-24T14:35:07.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7070000000000003e-06 2024-04-24T14:35:07.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980173+0000", 2024-04-24T14:35:07.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0957e-05 2024-04-24T14:35:07.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983381+0000", 2024-04-24T14:35:07.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032079930000000001 2024-04-24T14:35:07.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983628+0000", 2024-04-24T14:35:07.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024665700000000001 2024-04-24T14:35:07.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:07.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983964+0000", 2024-04-24T14:35:07.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000336332 2024-04-24T14:35:07.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:07.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992694+0000", 2024-04-24T14:35:07.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087297009999999994 2024-04-24T14:35:07.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:07.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994878+0000", 2024-04-24T14:35:07.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021841439999999998 2024-04-24T14:35:07.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:07.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994917+0000", 2024-04-24T14:35:07.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8544999999999998e-05 2024-04-24T14:35:07.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994947+0000", 2024-04-24T14:35:07.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9705000000000001e-05 2024-04-24T14:35:07.463 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.463 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.463 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.463 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:130 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:07.463 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:07.463 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.369538255999998, 2024-04-24T14:35:07.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014646289999999999, 2024-04-24T14:35:07.463 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.463 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:07.463 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.464 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:07.464 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:07.464 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 130 2024-04-24T14:35:07.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.464 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:07.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:07.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005844+0000", 2024-04-24T14:35:07.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:07.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005853+0000", 2024-04-24T14:35:07.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1319999999999994e-06 2024-04-24T14:35:07.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005870+0000", 2024-04-24T14:35:07.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7448999999999998e-05 2024-04-24T14:35:07.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005876+0000", 2024-04-24T14:35:07.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3319999999999997e-06 2024-04-24T14:35:07.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005913+0000", 2024-04-24T14:35:07.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6835999999999998e-05 2024-04-24T14:35:07.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006017+0000", 2024-04-24T14:35:07.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010358500000000001 2024-04-24T14:35:07.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:07.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006139+0000", 2024-04-24T14:35:07.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001227 2024-04-24T14:35:07.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:07.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.010194+0000", 2024-04-24T14:35:07.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040549380000000001 2024-04-24T14:35:07.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:07.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020365+0000", 2024-04-24T14:35:07.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010170983 2024-04-24T14:35:07.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:07.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020428+0000", 2024-04-24T14:35:07.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2457000000000002e-05 2024-04-24T14:35:07.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020492+0000", 2024-04-24T14:35:07.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4592999999999996e-05 2024-04-24T14:35:07.469 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.469 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.469 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.469 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:155 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1601536~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:07.470 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:07.470 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.335877469000003, 2024-04-24T14:35:07.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015144391, 2024-04-24T14:35:07.470 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.470 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:07.470 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.470 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:07.470 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:07.470 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 155 2024-04-24T14:35:07.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.470 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:07.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:07.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039506+0000", 2024-04-24T14:35:07.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:07.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039766+0000", 2024-04-24T14:35:07.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025994400000000002 2024-04-24T14:35:07.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039780+0000", 2024-04-24T14:35:07.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3366e-05 2024-04-24T14:35:07.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039787+0000", 2024-04-24T14:35:07.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4359999999999996e-06 2024-04-24T14:35:07.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.050886+0000", 2024-04-24T14:35:07.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011098993 2024-04-24T14:35:07.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051026+0000", 2024-04-24T14:35:07.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001395 2024-04-24T14:35:07.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:07.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051211+0000", 2024-04-24T14:35:07.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018547099999999999 2024-04-24T14:35:07.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:07.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054490+0000", 2024-04-24T14:35:07.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032785499999999999 2024-04-24T14:35:07.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:07.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054505+0000", 2024-04-24T14:35:07.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5906999999999999e-05 2024-04-24T14:35:07.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:07.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054594+0000", 2024-04-24T14:35:07.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8817e-05 2024-04-24T14:35:07.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054651+0000", 2024-04-24T14:35:07.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7040999999999998e-05 2024-04-24T14:35:07.475 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.475 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.475 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.476 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:156 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1732608~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:07.476 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:07.476 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.335586763000002, 2024-04-24T14:35:07.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016895194999999998, 2024-04-24T14:35:07.476 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.476 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:07.476 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.476 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:07.476 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:07.476 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 156 2024-04-24T14:35:07.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.477 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:07.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:07.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039797+0000", 2024-04-24T14:35:07.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040065+0000", 2024-04-24T14:35:07.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000268361 2024-04-24T14:35:07.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040086+0000", 2024-04-24T14:35:07.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1024000000000001e-05 2024-04-24T14:35:07.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040094+0000", 2024-04-24T14:35:07.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9379999999999992e-06 2024-04-24T14:35:07.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051936+0000", 2024-04-24T14:35:07.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011842518 2024-04-24T14:35:07.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052062+0000", 2024-04-24T14:35:07.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125482 2024-04-24T14:35:07.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:07.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052206+0000", 2024-04-24T14:35:07.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143953 2024-04-24T14:35:07.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:07.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054715+0000", 2024-04-24T14:35:07.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025088240000000002 2024-04-24T14:35:07.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:07.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056534+0000", 2024-04-24T14:35:07.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001819436 2024-04-24T14:35:07.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:07.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056610+0000", 2024-04-24T14:35:07.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5383000000000007e-05 2024-04-24T14:35:07.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056693+0000", 2024-04-24T14:35:07.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3283999999999997e-05 2024-04-24T14:35:07.482 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.482 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.482 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.482 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:157 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1863680~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:07.482 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:07.482 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.335279384000003, 2024-04-24T14:35:07.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016763674999999999, 2024-04-24T14:35:07.482 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.482 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:07.482 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.482 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:07.483 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:07.483 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157 2024-04-24T14:35:07.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.483 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:07.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:07.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040104+0000", 2024-04-24T14:35:07.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040244+0000", 2024-04-24T14:35:07.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014035499999999999 2024-04-24T14:35:07.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040255+0000", 2024-04-24T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0135999999999999e-05 2024-04-24T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040260+0000", 2024-04-24T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7919999999999998e-06 2024-04-24T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053406+0000", 2024-04-24T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013146023999999999 2024-04-24T14:35:07.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053531+0000", 2024-04-24T14:35:07.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012471800000000001 2024-04-24T14:35:07.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:07.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053715+0000", 2024-04-24T14:35:07.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018395099999999999 2024-04-24T14:35:07.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:07.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056711+0000", 2024-04-24T14:35:07.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029956520000000001 2024-04-24T14:35:07.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:07.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056783+0000", 2024-04-24T14:35:07.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2010999999999999e-05 2024-04-24T14:35:07.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:07.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056831+0000", 2024-04-24T14:35:07.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8189999999999998e-05 2024-04-24T14:35:07.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056869+0000", 2024-04-24T14:35:07.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7812000000000003e-05 2024-04-24T14:35:07.488 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.488 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.488 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.488 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:158 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1994752~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:07.488 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:07.488 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.335117457000003, 2024-04-24T14:35:07.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016845334999999999, 2024-04-24T14:35:07.489 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.489 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:07.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:07.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:07.489 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2024-04-24T14:35:07.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.489 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:07.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:07.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040266+0000", 2024-04-24T14:35:07.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040379+0000", 2024-04-24T14:35:07.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112728 2024-04-24T14:35:07.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040387+0000", 2024-04-24T14:35:07.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8499999999999994e-06 2024-04-24T14:35:07.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040391+0000", 2024-04-24T14:35:07.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2719999999999999e-06 2024-04-24T14:35:07.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054803+0000", 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014411548 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054982+0000", 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017941699999999999 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.055167+0000", 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018542100000000001 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:07.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056718+0000", 2024-04-24T14:35:07.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015505670000000001 2024-04-24T14:35:07.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:07.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057020+0000", 2024-04-24T14:35:07.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030227500000000002 2024-04-24T14:35:07.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:07.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057070+0000", 2024-04-24T14:35:07.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9611000000000003e-05 2024-04-24T14:35:07.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057112+0000", 2024-04-24T14:35:07.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2373999999999998e-05 2024-04-24T14:35:07.494 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.494 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.494 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.494 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:159 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:07.494 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:07.494 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.329682468999998, 2024-04-24T14:35:07.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015930650000000001, 2024-04-24T14:35:07.495 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.495 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:07.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:07.495 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2024-04-24T14:35:07.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.495 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:07.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:07.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045701+0000", 2024-04-24T14:35:07.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:07.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045707+0000", 2024-04-24T14:35:07.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.562e-06 2024-04-24T14:35:07.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045728+0000", 2024-04-24T14:35:07.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1109999999999999e-05 2024-04-24T14:35:07.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045738+0000", 2024-04-24T14:35:07.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0016e-05 2024-04-24T14:35:07.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056183+0000", 2024-04-24T14:35:07.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010444836000000001 2024-04-24T14:35:07.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056470+0000", 2024-04-24T14:35:07.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028696899999999998 2024-04-24T14:35:07.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057160+0000", 2024-04-24T14:35:07.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069004699999999995 2024-04-24T14:35:07.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057291+0000", 2024-04-24T14:35:07.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013096200000000001 2024-04-24T14:35:07.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.061633+0000", 2024-04-24T14:35:07.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004341414 2024-04-24T14:35:07.499 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.499 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.500 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.500 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:07.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:35:07.502 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:07.504+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:07.602 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:07.606+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:07.602 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:07.603 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:07.603 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:35:07.603 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:07.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:35:07.703 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:07.706+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:07.712 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:07.712 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:07.712 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:07.712 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:07.712 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:07.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:35:07.804 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:07.808+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:07.870 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.403137427000004, 2024-04-24T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.027780723999996, 2024-04-24T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.02635257, 2024-04-24T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.882 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:35:07.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.882 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:07.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:07.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:35:07.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:35:07.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:35:07.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:35:07.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:35:07.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:35:07.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:35:07.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:35:07.885 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.885 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.885 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.885 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.885 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:07.886 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.026325481000001, 2024-04-24T14:35:07.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:35:07.886 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.886 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.886 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.886 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.886 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.886 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:35:07.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.886 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.026301513999996, 2024-04-24T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.012317534999994, 2024-04-24T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.012201223999995, 2024-04-24T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.908 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.908 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:07.908 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.012153565999995, 2024-04-24T14:35:07.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:35:07.908 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.908 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.908 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.908 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.908 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.908 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:35:07.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.908 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:07.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:07.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:35:07.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:35:07.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:35:07.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:35:07.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:35:07.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:35:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:35:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:35:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:35:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:35:07.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:35:07.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:35:07.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:35:07.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:35:07.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:35:07.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:35:07.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:35:07.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:35:07.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:35:07.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:35:07.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:35:07.914 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.914 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.914 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.914 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.914 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:07.914 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.008116267999995, 2024-04-24T14:35:07.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:35:07.914 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.914 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.914 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.914 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.914 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.915 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:35:07.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.915 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:07.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:07.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:35:07.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:07.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:35:07.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:35:07.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:35:07.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:35:07.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:35:07.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:35:07.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:35:07.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:35:07.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:35:07.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:35:07.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:35:07.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:35:07.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:35:07.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:35:07.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:35:07.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:35:07.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:35:07.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:35:07.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:35:07.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:35:07.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.920 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.920 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.920 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:07.920 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.008052508999995, 2024-04-24T14:35:07.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:35:07.921 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.921 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.921 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:35:07.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.921 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:07.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:07.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:35:07.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:35:07.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:35:07.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:35:07.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:35:07.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:35:07.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:35:07.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:35:07.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:35:07.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:35:07.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:35:07.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:35:07.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:35:07.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:35:07.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:35:07.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:35:07.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:35:07.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:35:07.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:35:07.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:35:07.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:35:07.926 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.926 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.926 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.927 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.927 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:07.927 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.008027018999996, 2024-04-24T14:35:07.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:35:07.927 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.927 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.927 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:35:07.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.927 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:07.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:07.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:07.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:07.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:35:07.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:35:07.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:35:07.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:35:07.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:35:07.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:35:07.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:35:07.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:35:07.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:35:07.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:35:07.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:35:07.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:35:07.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:35:07.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:35:07.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:35:07.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:35:07.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:35:07.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:35:07.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.008003269, 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.934 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:35:07.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.934 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:07.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:07.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:35:07.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:35:07.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:35:07.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:35:07.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:35:07.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:35:07.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:35:07.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:35:07.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:35:07.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:35:07.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:35:07.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:35:07.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:35:07.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:35:07.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:35:07.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:35:07.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:35:07.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:35:07.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:35:07.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:35:07.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:35:07.939 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.939 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.939 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.939 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.939 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:07.940 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.007964756000007, 2024-04-24T14:35:07.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:35:07.940 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.940 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.940 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.940 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.940 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.940 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:35:07.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.940 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:07.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:07.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:35:07.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:07.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:35:07.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:35:07.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:35:07.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:35:07.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:35:07.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:35:07.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:35:07.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:35:07.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:35:07.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:35:07.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:35:07.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:35:07.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:35:07.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:35:07.945 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.945 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.945 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.945 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.946 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:07.946 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.007936559000001, 2024-04-24T14:35:07.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:35:07.946 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.946 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.946 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.946 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.946 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.946 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:35:07.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.946 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:35:07.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:35:07.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:35:07.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:35:07.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:35:07.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:35:07.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:35:07.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:35:07.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:35:07.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:35:07.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:35:07.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:35:07.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:35:07.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:35:07.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:35:07.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:35:07.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:35:07.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:35:07.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:35:07.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:35:07.951 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.951 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.952 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.952 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.952 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:07.952 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.007915101999998, 2024-04-24T14:35:07.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:35:07.952 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.952 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.952 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.952 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.952 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.952 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:35:07.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.953 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:07.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:07.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:35:07.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:07.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:35:07.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:35:07.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:35:07.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:35:07.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:35:07.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:35:07.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:35:07.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:35:07.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:35:07.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:35:07.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:35:07.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:35:07.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:35:07.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:35:07.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:35:07.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:35:07.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:35:07.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:35:07.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:35:07.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:35:07.958 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.958 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.958 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.958 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:07.958 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:07.958 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.007896259999995, 2024-04-24T14:35:07.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:35:07.958 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.959 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.959 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.959 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:07.959 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:07.959 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:35:07.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.959 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:07.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:07.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:35:07.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:07.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:35:07.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:35:07.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:35:07.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:35:07.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:35:07.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:35:07.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:35:07.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:35:07.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:35:07.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:35:07.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:35:07.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:35:07.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:35:07.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:35:07.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:35:07.964 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.964 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.964 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.964 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:07.964 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:07.964 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.795064111000002, 2024-04-24T14:35:07.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:35:07.965 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.965 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:07.965 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.965 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:35:07.965 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:07.965 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:35:07.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.965 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:07.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:07.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:35:07.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:07.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:35:07.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:35:07.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:35:07.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:35:07.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:35:07.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:35:07.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:35:07.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:35:07.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:07.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:35:07.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:35:07.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:35:07.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:35:07.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:07.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:35:07.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:35:07.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:35:07.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:35:07.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:35:07.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:35:07.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:35:07.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:35:07.970 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.970 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.970 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.970 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:39 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1753088~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 36.855291694999998, 2024-04-24T14:35:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01139394, 2024-04-24T14:35:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 39 2024-04-24T14:35:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018967+0000", 2024-04-24T14:35:07.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019242+0000", 2024-04-24T14:35:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027504500000000002 2024-04-24T14:35:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019274+0000", 2024-04-24T14:35:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2472999999999999e-05 2024-04-24T14:35:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019288+0000", 2024-04-24T14:35:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3283e-05 2024-04-24T14:35:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026753+0000", 2024-04-24T14:35:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074651120000000003 2024-04-24T14:35:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026971+0000", 2024-04-24T14:35:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021810800000000001 2024-04-24T14:35:07.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.027211+0000", 2024-04-24T14:35:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023975 2024-04-24T14:35:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030200+0000", 2024-04-24T14:35:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029890709999999998 2024-04-24T14:35:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:07.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030252+0000", 2024-04-24T14:35:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1926999999999999e-05 2024-04-24T14:35:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030312+0000", 2024-04-24T14:35:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0602e-05 2024-04-24T14:35:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030363+0000", 2024-04-24T14:35:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0342e-05 2024-04-24T14:35:07.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:40 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1884160~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 36.854955721000003, 2024-04-24T14:35:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011199064, 2024-04-24T14:35:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:07.977 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 40 2024-04-24T14:35:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:07.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019303+0000", 2024-04-24T14:35:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019729+0000", 2024-04-24T14:35:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042620899999999998 2024-04-24T14:35:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019769+0000", 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9847e-05 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019797+0000", 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8347000000000002e-05 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028299+0000", 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085010629999999997 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028385+0000", 2024-04-24T14:35:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6824999999999996e-05 2024-04-24T14:35:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028510+0000", 2024-04-24T14:35:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012483000000000001 2024-04-24T14:35:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:07.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030410+0000", 2024-04-24T14:35:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001899612 2024-04-24T14:35:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030416+0000", 2024-04-24T14:35:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7699999999999998e-06 2024-04-24T14:35:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030461+0000", 2024-04-24T14:35:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5157999999999997e-05 2024-04-24T14:35:07.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030504+0000", 2024-04-24T14:35:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2988999999999998e-05 2024-04-24T14:35:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:41 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 2015232~81920] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 36.854447014000002, 2024-04-24T14:35:07.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012345257, 2024-04-24T14:35:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 41 2024-04-24T14:35:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019812+0000", 2024-04-24T14:35:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:07.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019975+0000", 2024-04-24T14:35:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016325700000000001 2024-04-24T14:35:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020025+0000", 2024-04-24T14:35:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9484999999999998e-05 2024-04-24T14:35:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020035+0000", 2024-04-24T14:35:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0444e-05 2024-04-24T14:35:07.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029150+0000", 2024-04-24T14:35:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091150160000000001 2024-04-24T14:35:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029352+0000", 2024-04-24T14:35:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020166499999999999 2024-04-24T14:35:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029611+0000", 2024-04-24T14:35:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025949600000000001 2024-04-24T14:35:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030518+0000", 2024-04-24T14:35:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090633300000000005 2024-04-24T14:35:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032088+0000", 2024-04-24T14:35:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015703290000000001 2024-04-24T14:35:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032130+0000", 2024-04-24T14:35:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2273e-05 2024-04-24T14:35:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032159+0000", 2024-04-24T14:35:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8387000000000001e-05 2024-04-24T14:35:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:07.989 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:07.990 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:07.991 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:35:07.992 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:07.996+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:08.092 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:08.098+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:08.095 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd16809 2024-04-24T14:35:08.095 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '/entries inspected/ {print $1}' 2024-04-24T14:35:08.102 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:08.102 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:08.102 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:35:08.102 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:08.111 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:35:08.186 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:35:08.192+0000 7fc2e2599700 -1 JournalMetadata: 0x55581adcba70 failed to locate client: INSPECT 2024-04-24T14:35:08.192 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:08.196+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:08.195 INFO:tasks.workunit.client.0.smithi173.stderr:+ queue_length=4 2024-04-24T14:35:08.195 INFO:tasks.workunit.client.0.smithi173.stderr:+ tries=3 2024-04-24T14:35:08.196 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 3 -lt 10 ']' 2024-04-24T14:35:08.196 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 4 -gt 0 ']' 2024-04-24T14:35:08.196 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd16809 --verbose 2024-04-24T14:35:08.220 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:08.220 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:08.220 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:35:08.220 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:35:08.220 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:08.229 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:35:08.283 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=1 2024-04-24T14:35:08.284 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:08.284 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:08.284 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 0, 2024-04-24T14:35:08.284 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:08.284 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:08.284 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.362976+0000" 2024-04-24T14:35:08.284 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:08.284 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=2 2024-04-24T14:35:08.284 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:08.284 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:08.284 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 0, 2024-04-24T14:35:08.284 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:08.285 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:08.285 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.383412+0000" 2024-04-24T14:35:08.285 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:08.285 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=3 2024-04-24T14:35:08.285 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:08.285 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:08.285 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 4096, 2024-04-24T14:35:08.285 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:08.285 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:08.285 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.416941+0000" 2024-04-24T14:35:08.285 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:08.285 INFO:tasks.workunit.client.0.smithi173.stdout:Entry: tag_id=1, commit_tid=4 2024-04-24T14:35:08.286 INFO:tasks.workunit.client.0.smithi173.stdout:{ 2024-04-24T14:35:08.286 INFO:tasks.workunit.client.0.smithi173.stdout: "event_type": "AioDiscard", 2024-04-24T14:35:08.286 INFO:tasks.workunit.client.0.smithi173.stdout: "offset": 4096, 2024-04-24T14:35:08.286 INFO:tasks.workunit.client.0.smithi173.stdout: "length": 4096, 2024-04-24T14:35:08.286 INFO:tasks.workunit.client.0.smithi173.stdout: "discard_granularity_bytes": 4096, 2024-04-24T14:35:08.286 INFO:tasks.workunit.client.0.smithi173.stdout: "timestamp": "2024-04-24T14:35:04.439515+0000" 2024-04-24T14:35:08.286 INFO:tasks.workunit.client.0.smithi173.stdout:} 2024-04-24T14:35:08.286 INFO:tasks.workunit.client.0.smithi173.stdout:Summary: 2024-04-24T14:35:08.286 INFO:tasks.workunit.client.0.smithi173.stdout: 4 entries inspected, 0 errors 2024-04-24T14:35:08.293 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:35:08.296+0000 7fbd99462700 -1 JournalMetadata: 0x55fa43cf33a0 failed to locate client: INSPECT 2024-04-24T14:35:08.294 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:08.298+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:08.299 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 1 2024-04-24T14:35:08.308 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 1] 2024-04-24T14:35:08.308 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:35:08.308 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-24T14:35:08.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T14:35:08.333 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-24T14:35:08.335 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:35:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:35:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:35:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 165.713774094, 2024-04-24T14:35:08.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:35:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:35:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:08.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:35:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:35:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:35:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:35:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:35:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:35:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:35:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:35:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:35:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:35:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:35:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:08.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:35:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:35:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:35:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:35:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:35:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:35:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:35:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:35:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:35:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:35:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.341 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 165.713597007, 2024-04-24T14:35:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:35:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:35:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:35:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:08.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:35:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:35:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:35:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:35:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:35:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:35:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:35:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:35:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:35:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:35:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:35:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:35:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:35:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:35:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.347 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 165.71349309300001, 2024-04-24T14:35:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:35:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:35:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:35:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:35:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:35:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:35:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:35:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:35:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:35:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:35:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:35:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:35:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:35:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:35:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:35:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:35:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:35:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:35:08.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:35:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:35:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:35:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:35:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:35:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 165.71310464199999, 2024-04-24T14:35:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:35:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:35:08.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:35:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:35:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:35:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:35:08.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:35:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:35:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:35:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:35:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:35:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:35:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:35:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:35:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:35:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:35:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:35:08.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:35:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:35:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:35:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:35:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:35:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:35:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 165.71301943, 2024-04-24T14:35:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:35:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.360 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:35:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:35:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:35:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:35:08.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:35:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:35:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:35:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:35:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:35:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:35:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:35:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:35:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:35:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:35:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:35:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:35:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:35:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:35:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:08.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:35:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:35:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:35:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:35:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.366 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 165.712782549, 2024-04-24T14:35:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:35:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:35:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:35:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:35:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:35:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:35:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:35:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:35:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:35:08.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:35:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:35:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:35:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:35:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:08.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:35:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:35:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:35:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:35:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:35:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:35:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:35:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:35:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:35:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:35:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.372 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 165.71270102700001, 2024-04-24T14:35:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:35:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:35:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:35:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:35:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:35:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:35:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:35:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:35:08.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:35:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:35:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:35:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:35:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:35:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:08.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:35:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:35:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:35:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:35:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:35:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:35:08.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:35:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:35:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:35:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:35:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 165.28078868, 2024-04-24T14:35:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:35:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:35:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.379 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:35:08.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:35:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:35:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:35:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:35:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:35:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:35:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:35:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:35:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:35:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:35:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:35:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:35:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:35:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:35:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:35:08.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:35:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:35:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:35:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:35:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:35:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 165.28076038, 2024-04-24T14:35:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:35:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:35:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:35:08.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:35:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:35:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:35:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:35:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:35:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:35:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:35:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:35:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:35:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:35:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:35:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:35:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:35:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:35:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:35:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:35:08.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:35:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:35:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:35:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:35:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 165.28067267599999, 2024-04-24T14:35:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:35:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:35:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:35:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:35:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:35:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:35:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:35:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:35:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:35:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:35:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:35:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:35:08.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:35:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:35:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:35:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:35:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:35:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:35:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:35:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:35:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:35:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:35:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:35:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.396 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 165.28054987499999, 2024-04-24T14:35:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:35:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:08.397 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:35:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:35:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:35:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:35:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:35:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:35:08.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:35:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:35:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:35:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:35:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:35:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:35:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:35:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:35:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:35:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:35:08.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:35:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:35:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:35:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:35:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.402 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:35:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:35:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:35:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 95.974305314999995, 2024-04-24T14:35:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:35:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.403 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:35:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:35:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:35:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:35:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:35:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:35:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:35:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:35:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:35:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:35:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:08.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:35:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:35:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:35:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:35:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 88.280181315999997, 2024-04-24T14:35:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:35:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:35:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:35:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:35:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:08.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:35:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:35:08.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:35:08.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:35:08.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:35:08.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:35:08.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:35:08.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:35:08.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:35:08.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:08.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:35:08.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:35:08.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:35:08.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:35:08.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:08.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:35:08.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:35:08.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:35:08.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:35:08.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:35:08.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:35:08.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:35:08.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:35:08.414 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.414 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.414 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.415 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:08.415 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:08.415 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.320821350000003, 2024-04-24T14:35:08.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:35:08.415 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.415 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.415 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.415 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:08.415 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:08.415 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:35:08.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.415 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:08.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:08.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.416 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.416 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:35:08.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:08.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:35:08.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:35:08.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.417 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:35:08.417 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:35:08.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.417 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:35:08.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:35:08.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:35:08.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:35:08.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.418 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.418 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:08.418 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:35:08.418 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:35:08.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:35:08.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:35:08.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:08.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:35:08.419 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:35:08.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.419 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.419 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.419 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:35:08.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:35:08.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:35:08.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:35:08.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.420 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.420 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.420 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:35:08.420 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:35:08.420 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.421 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.421 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.421 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.421 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:08.421 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:08.421 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.320740795000006, 2024-04-24T14:35:08.421 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:35:08.421 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.421 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.421 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.421 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:08.422 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:08.422 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:35:08.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.422 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:08.422 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.422 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.422 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.422 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:08.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:35:08.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:08.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.423 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.423 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:35:08.423 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:35:08.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.423 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:35:08.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:35:08.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:35:08.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:35:08.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.424 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.424 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.424 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:35:08.424 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:35:08.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:08.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:35:08.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:35:08.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:35:08.425 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:35:08.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.425 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.425 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:08.425 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:35:08.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:35:08.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:35:08.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:35:08.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.426 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.426 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:35:08.426 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:35:08.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.426 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.427 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.427 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:35:08.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:35:08.427 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.427 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.427 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.427 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.427 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:08.427 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:08.427 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.320693066000004, 2024-04-24T14:35:08.427 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:35:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:35:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.428 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:08.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:35:08.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:08.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.429 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.429 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.429 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:35:08.429 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:35:08.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:35:08.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:35:08.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.430 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:35:08.430 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:35:08.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.430 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:35:08.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:35:08.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:08.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:35:08.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:35:08.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.431 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.431 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.431 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:35:08.431 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:35:08.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:08.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:35:08.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:35:08.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.432 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:35:08.432 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:35:08.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.432 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.432 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:08.433 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.320665302999998, 2024-04-24T14:35:08.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:35:08.434 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.434 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.434 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.434 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:08.434 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:08.434 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:35:08.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.434 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.434 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.434 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.434 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:08.434 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:08.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.435 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:35:08.435 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:08.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.435 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.435 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:35:08.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:35:08.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:35:08.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:35:08.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.436 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.436 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:35:08.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:35:08.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:35:08.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:35:08.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:08.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:35:08.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:35:08.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:35:08.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:35:08.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:08.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:35:08.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:35:08.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:35:08.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:35:08.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:35:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:35:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:35:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:35:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:08.440 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.320625867999993, 2024-04-24T14:35:08.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:35:08.440 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.440 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.440 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.440 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:08.440 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:08.440 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:35:08.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.440 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:08.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:08.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:35:08.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:08.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:35:08.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:35:08.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:35:08.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:35:08.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.442 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.442 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:35:08.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:35:08.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:35:08.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:35:08.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:08.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:35:08.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:35:08.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:35:08.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:35:08.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:08.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:35:08.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:35:08.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:35:08.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:35:08.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:35:08.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:35:08.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:35:08.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:35:08.445 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.445 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.445 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.445 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:08.445 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:08.446 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.320600884000001, 2024-04-24T14:35:08.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:35:08.446 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.446 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.446 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.446 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:08.446 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:08.446 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:35:08.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.446 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:08.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:08.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:35:08.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:08.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:35:08.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:35:08.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:35:08.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:35:08.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.448 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.448 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:35:08.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:35:08.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:35:08.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:35:08.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:08.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:35:08.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:35:08.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:35:08.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:35:08.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:08.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:35:08.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:35:08.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:35:08.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:35:08.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:35:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:35:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:35:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:35:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.452 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:08.452 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:08.452 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 69.320575224999999, 2024-04-24T14:35:08.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:35:08.452 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.452 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.452 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.452 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:08.452 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:08.452 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:35:08.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.452 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:08.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:08.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:35:08.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:08.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:35:08.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:35:08.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:35:08.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:35:08.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.454 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.454 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:35:08.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:35:08.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:35:08.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:35:08.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:08.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:35:08.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:35:08.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:35:08.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:35:08.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:08.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:35:08.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:35:08.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:35:08.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:35:08.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:35:08.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:35:08.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:35:08.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:35:08.457 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.457 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.457 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.458 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.458 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.458 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:08.460 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:35:08.460 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:08.464+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:08.560 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:08.564+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:08.597 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:08.597 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:08.597 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 110, 2024-04-24T14:35:08.597 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "6706935e-d09a-472b-b839-321319fa9e54", 2024-04-24T14:35:08.597 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2024-04-24T14:32:09.059925+0000", 2024-04-24T14:35:08.597 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2024-04-24T14:35:08.324729+0000", 2024-04-24T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2024-04-24T14:34:00.856741+0000", 2024-04-24T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2024-04-24T14:35:03.287210+0000", 2024-04-24T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5801984, 2024-04-24T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2024-04-24T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout: "noscrub", 2024-04-24T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2024-04-24T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2024-04-24T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2024-04-24T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2024-04-24T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 5, 2024-04-24T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2024-04-24T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 4, 2024-04-24T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 6, 2024-04-24T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2024-04-24T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "jewel", 2024-04-24T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "quincy", 2024-04-24T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2024-04-24T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2024-04-24T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2024-04-24T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:21.233022+0000", 2024-04-24T14:35:08.600 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-24T14:35:08.600 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-24T14:35:08.600 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:35:08.600 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:35:08.600 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:35:08.600 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:35:08.600 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:35:08.600 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:35:08.600 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:35:08.600 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:35:08.600 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:35:08.600 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:35:08.600 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2024-04-24T14:35:08.601 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2024-04-24T14:35:08.601 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2024-04-24T14:35:08.601 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2024-04-24T14:35:08.601 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2024-04-24T14:35:08.601 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:35:08.601 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:35:08.601 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:35:08.601 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:35:08.601 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:35:08.601 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:35:08.601 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:35:08.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.601 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "14", 2024-04-24T14:35:08.601 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-24T14:35:08.602 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T14:35:08.602 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T14:35:08.602 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:35:08.602 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:35:08.602 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-24T14:35:08.602 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-24T14:35:08.602 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:35:08.602 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:35:08.602 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:35:08.602 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:35:08.602 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:35:08.602 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:35:08.602 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:35:08.603 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:35:08.603 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:35:08.603 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:35:08.603 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:35:08.603 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:35:08.603 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:35:08.603 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:35:08.603 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:35:08.603 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:35:08.603 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:35:08.603 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:35:08.603 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:35:08.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.604 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:35:08.604 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:35:08.604 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:35:08.604 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:35:08.604 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:35:08.604 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:35:08.604 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:35:08.604 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:35:08.604 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:35:08.604 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:35:08.604 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:35:08.604 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-24T14:35:08.604 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2024-04-24T14:35:08.605 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2024-04-24T14:35:08.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.605 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:35:08.605 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2024-04-24T14:35:08.605 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.605 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-24T14:35:08.605 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2024-04-24T14:35:08.605 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:26.609964+0000", 2024-04-24T14:35:08.605 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-24T14:35:08.605 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T14:35:08.605 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:35:08.606 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:35:08.606 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:35:08.606 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:35:08.606 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:35:08.606 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:35:08.606 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:35:08.606 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:35:08.606 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:35:08.606 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:35:08.606 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2024-04-24T14:35:08.606 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2024-04-24T14:35:08.606 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2024-04-24T14:35:08.607 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2024-04-24T14:35:08.607 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2024-04-24T14:35:08.607 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:35:08.607 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:35:08.607 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:35:08.607 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:35:08.607 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:35:08.607 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:35:08.607 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:35:08.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.607 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "89", 2024-04-24T14:35:08.607 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "21", 2024-04-24T14:35:08.607 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "21", 2024-04-24T14:35:08.607 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "21", 2024-04-24T14:35:08.608 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:35:08.608 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:35:08.608 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 8, 2024-04-24T14:35:08.608 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 89, 2024-04-24T14:35:08.608 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:35:08.608 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:35:08.608 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:35:08.608 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:35:08.608 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [ 2024-04-24T14:35:08.608 INFO:teuthology.orchestra.run.smithi090.stdout: 3 2024-04-24T14:35:08.608 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:08.608 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:35:08.608 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": 3, 2024-04-24T14:35:08.609 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": 3, 2024-04-24T14:35:08.609 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:35:08.609 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:35:08.609 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:35:08.609 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:35:08.609 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:35:08.609 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:35:08.609 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:35:08.609 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:35:08.609 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:35:08.609 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:35:08.609 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:35:08.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.609 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:35:08.610 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:35:08.610 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:35:08.610 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:35:08.610 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:35:08.610 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:35:08.610 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:35:08.610 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:35:08.610 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:35:08.610 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:35:08.610 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:35:08.610 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:35:08.610 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:35:08.610 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:35:08.611 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.611 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-24T14:35:08.611 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "cache", 2024-04-24T14:35:08.611 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:33.987564+0000", 2024-04-24T14:35:08.611 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8201, 2024-04-24T14:35:08.611 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-04-24T14:35:08.611 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:35:08.611 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:35:08.611 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:35:08.611 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:35:08.611 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:35:08.612 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:35:08.612 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:35:08.612 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:35:08.612 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:35:08.612 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:35:08.612 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 4, 2024-04-24T14:35:08.612 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 4, 2024-04-24T14:35:08.612 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 4, 2024-04-24T14:35:08.612 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 4, 2024-04-24T14:35:08.612 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 4, 2024-04-24T14:35:08.612 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:35:08.612 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:35:08.612 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:35:08.612 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "89", 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "21", 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "21", 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "21", 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 8, 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 89, 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:35:08.613 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:35:08.614 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:35:08.614 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:35:08.614 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": 2, 2024-04-24T14:35:08.614 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:35:08.614 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:35:08.614 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "writeback", 2024-04-24T14:35:08.614 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:35:08.614 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 250, 2024-04-24T14:35:08.614 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:35:08.614 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:35:08.614 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:35:08.614 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:35:08.614 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:35:08.615 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:35:08.615 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:35:08.615 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bloom", 2024-04-24T14:35:08.615 INFO:teuthology.orchestra.run.smithi090.stdout: "false_positive_probability": 0.050000000000000003, 2024-04-24T14:35:08.615 INFO:teuthology.orchestra.run.smithi090.stdout: "target_size": 0, 2024-04-24T14:35:08.615 INFO:teuthology.orchestra.run.smithi090.stdout: "seed": 0 2024-04-24T14:35:08.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.615 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 60, 2024-04-24T14:35:08.615 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 8, 2024-04-24T14:35:08.615 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:35:08.615 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:35:08.615 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:35:08.615 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:35:08.615 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:35:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [ 2024-04-24T14:35:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:35:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:35:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:35:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:35:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:35:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:35:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:35:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000 2024-04-24T14:35:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:35:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:35:08.616 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:35:08.617 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:35:08.617 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:35:08.617 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:35:08.617 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.617 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 4, 2024-04-24T14:35:08.617 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "new_default_pool16809", 2024-04-24T14:35:08.617 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:33:35.603212+0000", 2024-04-24T14:35:08.617 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-24T14:35:08.617 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T14:35:08.617 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:35:08.617 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:35:08.617 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:35:08.618 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:35:08.618 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:35:08.618 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:35:08.618 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:35:08.618 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:35:08.618 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:35:08.618 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:35:08.618 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2024-04-24T14:35:08.618 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2024-04-24T14:35:08.618 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2024-04-24T14:35:08.618 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2024-04-24T14:35:08.618 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2024-04-24T14:35:08.618 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:35:08.619 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:35:08.619 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:35:08.619 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:35:08.619 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:35:08.619 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:35:08.619 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:35:08.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.619 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "53", 2024-04-24T14:35:08.619 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-24T14:35:08.619 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T14:35:08.619 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T14:35:08.619 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:35:08.619 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:35:08.619 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 3, 2024-04-24T14:35:08.620 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 53, 2024-04-24T14:35:08.620 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:35:08.620 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:35:08.620 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:35:08.620 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:35:08.620 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:35:08.620 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:35:08.620 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:35:08.620 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:35:08.620 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:35:08.620 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:35:08.620 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:35:08.620 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:35:08.621 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:35:08.621 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:35:08.621 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:35:08.621 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:35:08.621 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:35:08.621 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:35:08.621 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:35:08.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.621 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:35:08.621 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:35:08.621 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:35:08.621 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:35:08.621 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:35:08.622 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:35:08.622 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:35:08.622 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:35:08.622 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:35:08.622 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:35:08.622 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:35:08.622 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:35:08.622 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:35:08.622 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:35:08.622 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.622 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.622 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:08.622 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-24T14:35:08.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.623 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-24T14:35:08.623 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "a2506b4f-3486-4f51-972f-1d9e4b015346", 2024-04-24T14:35:08.623 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:35:08.623 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:35:08.623 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:35:08.623 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.484344482421875, 2024-04-24T14:35:08.623 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:35:08.623 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:35:08.623 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:35:08.623 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 96, 2024-04-24T14:35:08.623 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:35:08.624 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:35:08.624 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:35:08.624 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.624 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.624 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2024-04-24T14:35:08.624 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:08.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.624 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.624 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2024-04-24T14:35:08.624 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:08.624 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.624 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.625 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:35:08.625 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.625 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.625 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2024-04-24T14:35:08.625 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:08.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.625 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.625 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2024-04-24T14:35:08.625 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:08.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.626 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.626 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:35:08.626 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.626 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.626 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2024-04-24T14:35:08.626 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:08.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.626 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.627 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2024-04-24T14:35:08.627 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:08.627 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.627 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.627 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:35:08.627 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.627 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2024-04-24T14:35:08.627 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:08.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.628 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2024-04-24T14:35:08.628 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:08.628 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.628 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.628 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6817/20309", 2024-04-24T14:35:08.628 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6819/20309", 2024-04-24T14:35:08.628 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/20309", 2024-04-24T14:35:08.628 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6821/20309", 2024-04-24T14:35:08.628 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:35:08.628 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:35:08.628 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:35:08.628 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.629 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-24T14:35:08.629 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "9e17324f-a615-494f-ba0d-7c8aa8dac95c", 2024-04-24T14:35:08.629 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:35:08.629 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2024-04-24T14:35:08.629 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2024-04-24T14:35:08.629 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.2843475341796875, 2024-04-24T14:35:08.629 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 11, 2024-04-24T14:35:08.629 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 31, 2024-04-24T14:35:08.629 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 66, 2024-04-24T14:35:08.629 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 93, 2024-04-24T14:35:08.629 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 32, 2024-04-24T14:35:08.629 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:35:08.630 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:35:08.630 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.630 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.630 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2024-04-24T14:35:08.630 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:08.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.630 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.630 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2024-04-24T14:35:08.630 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:08.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.630 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.631 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:35:08.631 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.631 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.631 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2024-04-24T14:35:08.631 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:08.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.631 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.631 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2024-04-24T14:35:08.631 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:08.631 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.631 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.632 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:35:08.632 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.632 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.632 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2024-04-24T14:35:08.632 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:08.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.632 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.632 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2024-04-24T14:35:08.632 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:08.632 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.632 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.633 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:35:08.633 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.633 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.633 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2024-04-24T14:35:08.633 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:08.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.633 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.633 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2024-04-24T14:35:08.633 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:08.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.633 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.634 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/34413", 2024-04-24T14:35:08.634 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/34413", 2024-04-24T14:35:08.634 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/34413", 2024-04-24T14:35:08.634 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/34413", 2024-04-24T14:35:08.634 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:35:08.634 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:35:08.634 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:35:08.634 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.634 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-24T14:35:08.635 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7", 2024-04-24T14:35:08.635 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:35:08.635 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:35:08.635 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:35:08.635 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.8896331787109375, 2024-04-24T14:35:08.635 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:35:08.635 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:35:08.635 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:35:08.635 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 99, 2024-04-24T14:35:08.635 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:35:08.635 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:35:08.635 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:35:08.635 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.636 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.636 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2024-04-24T14:35:08.636 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:08.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.636 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.636 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2024-04-24T14:35:08.636 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:08.636 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.636 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.636 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:35:08.636 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2024-04-24T14:35:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2024-04-24T14:35:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:35:08.637 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.638 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.638 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2024-04-24T14:35:08.638 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:08.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.638 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.638 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2024-04-24T14:35:08.638 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:08.638 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.638 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.638 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:35:08.639 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.639 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.639 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2024-04-24T14:35:08.639 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:08.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.639 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.639 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2024-04-24T14:35:08.639 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:08.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.639 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.639 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6809/20310", 2024-04-24T14:35:08.640 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6811/20310", 2024-04-24T14:35:08.640 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6815/20310", 2024-04-24T14:35:08.640 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6813/20310", 2024-04-24T14:35:08.640 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:35:08.640 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:35:08.640 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:35:08.640 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.640 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-24T14:35:08.640 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "509c16da-22b2-48e8-aa51-c320c8c4fd8b", 2024-04-24T14:35:08.640 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:35:08.640 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:35:08.641 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:35:08.641 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:35:08.641 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:35:08.641 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:35:08.641 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:35:08.641 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 107, 2024-04-24T14:35:08.641 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:35:08.641 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:35:08.641 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:35:08.641 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.641 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.641 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6800", 2024-04-24T14:35:08.641 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6801", 2024-04-24T14:35:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:35:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.642 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.643 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6802", 2024-04-24T14:35:08.643 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:08.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.643 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6803", 2024-04-24T14:35:08.643 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:08.643 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.643 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.643 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:35:08.643 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.644 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.644 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6806", 2024-04-24T14:35:08.644 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:08.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.644 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.644 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6807", 2024-04-24T14:35:08.644 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:08.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.644 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.644 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:35:08.644 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.645 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.645 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6804", 2024-04-24T14:35:08.645 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:08.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.645 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.645 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6805", 2024-04-24T14:35:08.645 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:08.645 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.645 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.646 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6801/20316", 2024-04-24T14:35:08.646 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6803/20316", 2024-04-24T14:35:08.646 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6807/20316", 2024-04-24T14:35:08.646 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6805/20316", 2024-04-24T14:35:08.646 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:35:08.646 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:35:08.646 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:35:08.646 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.646 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-24T14:35:08.646 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1462066b-4e59-46d4-9b85-1f61ee1d56a6", 2024-04-24T14:35:08.646 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:35:08.647 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:35:08.647 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:35:08.647 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:35:08.647 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:35:08.647 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:35:08.647 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:35:08.647 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 107, 2024-04-24T14:35:08.647 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:35:08.647 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:35:08.647 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:35:08.647 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.647 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.647 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6808", 2024-04-24T14:35:08.648 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:08.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.648 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.648 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6809", 2024-04-24T14:35:08.648 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:08.648 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.648 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.648 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:35:08.648 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.648 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.649 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6810", 2024-04-24T14:35:08.649 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:08.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.649 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.649 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6811", 2024-04-24T14:35:08.649 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:08.649 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.649 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.649 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:35:08.649 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.649 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.650 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6814", 2024-04-24T14:35:08.650 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:08.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.650 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.650 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6815", 2024-04-24T14:35:08.650 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:08.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.650 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.650 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:35:08.650 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.650 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.651 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6812", 2024-04-24T14:35:08.651 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:08.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.651 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.651 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6813", 2024-04-24T14:35:08.651 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:08.651 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.651 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.651 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6809/20318", 2024-04-24T14:35:08.651 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6811/20318", 2024-04-24T14:35:08.651 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6815/20318", 2024-04-24T14:35:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6813/20318", 2024-04-24T14:35:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:35:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:35:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:35:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-24T14:35:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "4dc954a8-5f3c-4826-b9bc-98cf30519382", 2024-04-24T14:35:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:35:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:35:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:35:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:35:08.652 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:35:08.653 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:35:08.653 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:35:08.653 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 107, 2024-04-24T14:35:08.653 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:35:08.653 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:35:08.653 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:35:08.653 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.653 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.653 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6816", 2024-04-24T14:35:08.653 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:08.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.654 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.654 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6817", 2024-04-24T14:35:08.654 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:08.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.654 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.654 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:35:08.654 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.654 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.654 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6818", 2024-04-24T14:35:08.654 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:08.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.655 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.655 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6819", 2024-04-24T14:35:08.655 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:08.655 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.655 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.655 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:35:08.655 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.655 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.655 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6822", 2024-04-24T14:35:08.655 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:08.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.656 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.656 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6823", 2024-04-24T14:35:08.656 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:08.656 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.656 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.656 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:35:08.656 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:08.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.656 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:08.656 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6820", 2024-04-24T14:35:08.656 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:08.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.657 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:08.657 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6821", 2024-04-24T14:35:08.657 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:08.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.657 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.657 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6817/20317", 2024-04-24T14:35:08.657 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6819/20317", 2024-04-24T14:35:08.657 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6823/20317", 2024-04-24T14:35:08.657 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6821/20317", 2024-04-24T14:35:08.658 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:35:08.658 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:35:08.658 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:35:08.658 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:08.658 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.658 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:08.658 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2024-04-24T14:35:08.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.658 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-24T14:35:08.658 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:35:08.658 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:35:08.658 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:35:08.658 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:35:08.658 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:35:08.659 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.792899+0000", 2024-04-24T14:35:08.659 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:35:08.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.659 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-24T14:35:08.659 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:35:08.659 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:35:08.659 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:35:08.659 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:35:08.659 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2024-04-24T14:35:08.659 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.580414+0000", 2024-04-24T14:35:08.659 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 32 2024-04-24T14:35:08.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.660 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-24T14:35:08.660 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:35:08.660 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:35:08.660 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:35:08.660 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:35:08.660 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:35:08.660 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.713336+0000", 2024-04-24T14:35:08.660 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:35:08.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.660 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-24T14:35:08.661 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:35:08.661 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:35:08.661 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:35:08.661 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:35:08.661 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:35:08.661 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.579638+0000", 2024-04-24T14:35:08.661 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:35:08.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.661 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-24T14:35:08.661 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:35:08.661 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:35:08.662 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:35:08.662 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:35:08.662 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:35:08.662 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.570510+0000", 2024-04-24T14:35:08.662 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:35:08.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:08.662 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-24T14:35:08.662 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:35:08.662 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:35:08.662 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:35:08.662 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:35:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:35:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.736558+0000", 2024-04-24T14:35:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:35:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2024-04-24T14:35:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [], 2024-04-24T14:35:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2024-04-24T14:35:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2024-04-24T14:35:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2024-04-24T14:35:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2024-04-24T14:35:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2024-04-24T14:35:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2024-04-24T14:35:08.663 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-24T14:35:08.664 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-24T14:35:08.664 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-24T14:35:08.664 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-24T14:35:08.664 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2024-04-24T14:35:08.664 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:08.664 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [], 2024-04-24T14:35:08.664 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2024-04-24T14:35:08.664 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2024-04-24T14:35:08.664 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2024-04-24T14:35:08.664 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2024-04-24T14:35:08.664 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2024-04-24T14:35:08.664 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2024-04-24T14:35:08.664 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2024-04-24T14:35:08.665 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2024-04-24T14:35:08.665 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2024-04-24T14:35:08.665 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2024-04-24T14:35:08.665 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:08.665 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:08.667 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:08.667 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:08.667 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:35:08.667 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:08.668 INFO:tasks.thrashosds.thrasher:j is {'epoch': 110, 'fsid': '6706935e-d09a-472b-b839-321319fa9e54', 'created': '2024-04-24T14:32:09.059925+0000', 'modified': '2024-04-24T14:35:08.324729+0000', 'last_up_change': '2024-04-24T14:34:00.856741+0000', 'last_in_change': '2024-04-24T14:35:03.287210+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T14:32:21.233022+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T14:32:26.609964+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '21', 'last_force_op_resend_prenautilus': '21', 'last_force_op_resend_preluminous': '21', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 8, 'snap_epoch': 89, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-04-24T14:32:33.987564+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '21', 'last_force_op_resend_prenautilus': '21', 'last_force_op_resend_preluminous': '21', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 8, 'snap_epoch': 89, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 2, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'new_default_pool16809', 'create_time': '2024-04-24T14:33:35.603212+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 53, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'a2506b4f-3486-4f51-972f-1d9e4b015346', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.484344482421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 20309}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6819', 'nonce': 20309}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 20309}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6820', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 20309}]}, 'public_addr': '172.21.15.90:6817/20309', 'cluster_addr': '172.21.15.90:6819/20309', 'heartbeat_back_addr': '172.21.15.90:6823/20309', 'heartbeat_front_addr': '172.21.15.90:6821/20309', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9e17324f-a615-494f-ba0d-7c8aa8dac95c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.2843475341796875, 'last_clean_begin': 11, 'last_clean_end': 31, 'up_from': 66, 'up_thru': 93, 'down_at': 32, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 34413}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 34413}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 34413}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 34413}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 34413}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 34413}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 34413}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 34413}]}, 'public_addr': '172.21.15.90:6801/34413', 'cluster_addr': '172.21.15.90:6803/34413', 'heartbeat_back_addr': '172.21.15.90:6807/34413', 'heartbeat_front_addr': '172.21.15.90:6805/34413', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8896331787109375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 20310}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6810', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 20310}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6814', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 20310}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 20310}]}, 'public_addr': '172.21.15.90:6809/20310', 'cluster_addr': '172.21.15.90:6811/20310', 'heartbeat_back_addr': '172.21.15.90:6815/20310', 'heartbeat_front_addr': '172.21.15.90:6813/20310', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '509c16da-22b2-48e8-aa51-c320c8c4fd8b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6800', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6801', 'nonce': 20316}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6802', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6803', 'nonce': 20316}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6806', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6807', 'nonce': 20316}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6804', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6805', 'nonce': 20316}]}, 'public_addr': '172.21.15.103:6801/20316', 'cluster_addr': '172.21.15.103:6803/20316', 'heartbeat_back_addr': '172.21.15.103:6807/20316', 'heartbeat_front_addr': '172.21.15.103:6805/20316', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1462066b-4e59-46d4-9b85-1f61ee1d56a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6808', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6809', 'nonce': 20318}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6810', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6811', 'nonce': 20318}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6814', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6815', 'nonce': 20318}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6812', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6813', 'nonce': 20318}]}, 'public_addr': '172.21.15.103:6809/20318', 'cluster_addr': '172.21.15.103:6811/20318', 'heartbeat_back_addr': '172.21.15.103:6815/20318', 'heartbeat_front_addr': '172.21.15.103:6813/20318', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4dc954a8-5f3c-4826-b9bc-98cf30519382', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6816', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6817', 'nonce': 20317}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6818', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6819', 'nonce': 20317}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6822', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6823', 'nonce': 20317}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6820', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6821', 'nonce': 20317}]}, 'public_addr': '172.21.15.103:6817/20317', 'cluster_addr': '172.21.15.103:6819/20317', 'heartbeat_back_addr': '172.21.15.103:6823/20317', 'heartbeat_front_addr': '172.21.15.103:6821/20317', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.792899+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.580414+0000', 'dead_epoch': 32}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.713336+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.579638+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.570510+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.736558+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T14:35:08.668 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:35:08.669 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:35:08.669 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:08.668+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:08.766 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:08.768+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:08.817 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:08.818 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:08.818 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:35:08.818 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:35:08.818 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:08.827 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:35:08.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:35:08.890 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:08.894+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:08.930 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:08.930 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:35:08.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:35:08.930 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:35:08.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.269647155000001, 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:35:08.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:35:08.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:35:08.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:35:08.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:35:08.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:35:08.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:35:08.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:35:08.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:383 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:08.935 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.266612628999994, 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076536540000000002, 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 383 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667349+0000", 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667352+0000", 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.692e-06 2024-04-24T14:35:08.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667362+0000", 2024-04-24T14:35:08.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8139999999999995e-06 2024-04-24T14:35:08.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667368+0000", 2024-04-24T14:35:08.938 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8560000000000003e-06 2024-04-24T14:35:08.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.938 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.938 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.938 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674102+0000", 2024-04-24T14:35:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067344340000000001 2024-04-24T14:35:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.674246+0000", 2024-04-24T14:35:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000143392 2024-04-24T14:35:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675003+0000", 2024-04-24T14:35:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00075693799999999997 2024-04-24T14:35:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.940 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:08.940 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:08.940 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.266588443000003, 2024-04-24T14:35:08.940 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:35:08.940 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.940 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.940 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.940 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:08.940 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:08.940 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:35:08.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.941 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:08.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.941 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:08.941 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.941 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.941 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:08.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:08.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:35:08.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:35:08.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.942 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.942 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:35:08.942 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:35:08.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.942 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.943 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.944 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.944 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:35:08.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:35:08.944 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.944 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.944 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.944 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:08.944 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:35:08.944 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.266571059, 2024-04-24T14:35:08.944 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:35:08.944 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.944 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.945 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.945 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:08.945 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:08.945 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:35:08.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.945 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:35:08.945 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.945 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.945 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.945 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.945 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.946 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:35:08.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:35:08.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:35:08.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:35:08.948 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.948 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.948 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.948 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:08.948 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:35:08.948 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.252134967000003, 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:35:08.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:35:08.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:35:08.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:35:08.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.951 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.951 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:35:08.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:35:08.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:35:08.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:35:08.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:35:08.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:35:08.952 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.952 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.952 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.953 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:08.953 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:35:08.953 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.252102668000006, 2024-04-24T14:35:08.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:35:08.953 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.953 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.953 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.953 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:08.953 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:08.953 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:35:08.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.953 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:35:08.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:35:08.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:35:08.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:08.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:35:08.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:08.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:35:08.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:35:08.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:08.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:35:08.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:35:08.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:08.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:35:08.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:415 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.237850390999995, 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076371169999999997, 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 415 2024-04-24T14:35:08.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.958 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:08.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:08.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:35:08.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:08.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696111+0000", 2024-04-24T14:35:08.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:08.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:08.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696110+0000", 2024-04-24T14:35:08.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:08.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:08.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696116+0000", 2024-04-24T14:35:08.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4940000000000003e-06 2024-04-24T14:35:08.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:08.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:08.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696129+0000", 2024-04-24T14:35:08.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2945000000000001e-05 2024-04-24T14:35:08.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:08.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696134+0000", 2024-04-24T14:35:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0889999999999999e-06 2024-04-24T14:35:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703074+0000", 2024-04-24T14:35:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0069402509999999997 2024-04-24T14:35:09.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703164+0000", 2024-04-24T14:35:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0334000000000006e-05 2024-04-24T14:35:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703748+0000", 2024-04-24T14:35:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058391200000000002 2024-04-24T14:35:09.090 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.237821291000003, 2024-04-24T14:35:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:35:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:09.091 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:35:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:09.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:35:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:35:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:35:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:35:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:35:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:35:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:35:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:35:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:35:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:35:09.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:35:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:35:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.237795360999996, 2024-04-24T14:35:09.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:35:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:35:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:35:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:35:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:35:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:35:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:35:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:35:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:35:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:35:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:35:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:35:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:35:09.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:35:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:431 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:35:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.222074896999999, 2024-04-24T14:35:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077801820000000001, 2024-04-24T14:35:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:09.101 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 431 2024-04-24T14:35:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:35:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711887+0000", 2024-04-24T14:35:09.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711886+0000", 2024-04-24T14:35:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711891+0000", 2024-04-24T14:35:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8770000000000001e-06 2024-04-24T14:35:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711905+0000", 2024-04-24T14:35:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4299e-05 2024-04-24T14:35:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711911+0000", 2024-04-24T14:35:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0730000000000001e-06 2024-04-24T14:35:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718701+0000", 2024-04-24T14:35:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0067902639999999998 2024-04-24T14:35:09.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.718893+0000", 2024-04-24T14:35:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000191373 2024-04-24T14:35:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719667+0000", 2024-04-24T14:35:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077435999999999998 2024-04-24T14:35:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:35:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.222042965, 2024-04-24T14:35:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:35:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:35:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:35:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:09.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:35:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:35:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:09.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:35:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:35:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:35:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:35:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:35:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:35:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:35:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:35:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:35:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:35:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:35:09.109 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:35:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.221975448999999, 2024-04-24T14:35:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:35:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:35:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:09.110 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:35:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:35:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:35:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:35:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:35:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:35:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:35:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:35:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:35:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:35:09.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:35:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:35:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.164562566000001, 2024-04-24T14:35:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:35:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:09.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:09.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:35:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:35:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:35:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:35:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:35:09.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:35:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:35:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:35:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:35:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:35:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:35:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:35:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:35:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:35:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.118 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:103 3.1 e87/74 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head v 87'261, mlcod=87'258)", 2024-04-24T14:35:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:35:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.973281362000002, 2024-04-24T14:35:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019745728000000001, 2024-04-24T14:35:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:35:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.119 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:35:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960679+0000", 2024-04-24T14:35:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960692+0000", 2024-04-24T14:35:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3709999999999999e-05 2024-04-24T14:35:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960722+0000", 2024-04-24T14:35:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9791999999999999e-05 2024-04-24T14:35:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960733+0000", 2024-04-24T14:35:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.128e-05 2024-04-24T14:35:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960857+0000", 2024-04-24T14:35:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000123981 2024-04-24T14:35:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960926+0000", 2024-04-24T14:35:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8881999999999995e-05 2024-04-24T14:35:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980353+0000", 2024-04-24T14:35:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019426853000000001 2024-04-24T14:35:09.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980426+0000", 2024-04-24T14:35:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2803000000000001e-05 2024-04-24T14:35:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:105 3.1 e87/74 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head v 87'263, mlcod=87'258)", 2024-04-24T14:35:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:35:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.964154827999998, 2024-04-24T14:35:09.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014864743999999999, 2024-04-24T14:35:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:35:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:35:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969805+0000", 2024-04-24T14:35:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969820+0000", 2024-04-24T14:35:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5116999999999999e-05 2024-04-24T14:35:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969852+0000", 2024-04-24T14:35:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2135999999999999e-05 2024-04-24T14:35:09.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969864+0000", 2024-04-24T14:35:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.189e-05 2024-04-24T14:35:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972122+0000", 2024-04-24T14:35:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022580389999999999 2024-04-24T14:35:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972165+0000", 2024-04-24T14:35:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2429000000000003e-05 2024-04-24T14:35:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984619+0000", 2024-04-24T14:35:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012454735 2024-04-24T14:35:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984672+0000", 2024-04-24T14:35:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2077999999999998e-05 2024-04-24T14:35:09.127 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.128 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.128 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.128 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:112 3.1 e87/74 3:a983798d:::rbd_data.12ff64619e2d.0000000000000006:head v 87'265, mlcod=87'258)", 2024-04-24T14:35:09.128 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:35:09.128 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.953815992000003, 2024-04-24T14:35:09.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012821978, 2024-04-24T14:35:09.128 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:09.128 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:09.128 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:09.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:09.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:35:09.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:09.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:35:09.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:09.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980144+0000", 2024-04-24T14:35:09.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:09.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:09.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980159+0000", 2024-04-24T14:35:09.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4769999999999999e-05 2024-04-24T14:35:09.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:09.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980189+0000", 2024-04-24T14:35:09.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0419000000000001e-05 2024-04-24T14:35:09.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:09.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980213+0000", 2024-04-24T14:35:09.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4036000000000002e-05 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980452+0000", 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00023921500000000001 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980468+0000", 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5843999999999999e-05 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992932+0000", 2024-04-24T14:35:09.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012463314 2024-04-24T14:35:09.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:09.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992968+0000", 2024-04-24T14:35:09.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6075999999999998e-05 2024-04-24T14:35:09.132 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.132 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.132 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.132 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:113 3.1 e87/74 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head v 87'267, mlcod=87'258)", 2024-04-24T14:35:09.132 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:35:09.132 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.950281267999998, 2024-04-24T14:35:09.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010561022999999999, 2024-04-24T14:35:09.133 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:09.133 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:09.133 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:09.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:09.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:35:09.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:09.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:35:09.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:09.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983679+0000", 2024-04-24T14:35:09.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:09.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:09.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983699+0000", 2024-04-24T14:35:09.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0392000000000001e-05 2024-04-24T14:35:09.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:09.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983737+0000", 2024-04-24T14:35:09.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8340000000000002e-05 2024-04-24T14:35:09.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:09.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983756+0000", 2024-04-24T14:35:09.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8402000000000001e-05 2024-04-24T14:35:09.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:09.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984706+0000", 2024-04-24T14:35:09.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095024400000000005 2024-04-24T14:35:09.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:09.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984741+0000", 2024-04-24T14:35:09.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4721000000000002e-05 2024-04-24T14:35:09.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:09.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994211+0000", 2024-04-24T14:35:09.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094708559999999997 2024-04-24T14:35:09.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:09.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994241+0000", 2024-04-24T14:35:09.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9941e-05 2024-04-24T14:35:09.136 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.137 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.137 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.137 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:115 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:09.137 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:35:09.137 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.949827231, 2024-04-24T14:35:09.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010347242, 2024-04-24T14:35:09.137 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:09.137 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:09.137 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:09.137 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:35:09.138 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:09.138 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:35:09.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.138 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:09.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:09.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:35:09.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:09.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:35:09.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:09.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984132+0000", 2024-04-24T14:35:09.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:09.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:09.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984142+0000", 2024-04-24T14:35:09.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7370000000000001e-06 2024-04-24T14:35:09.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984154+0000", 2024-04-24T14:35:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2288e-05 2024-04-24T14:35:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984163+0000", 2024-04-24T14:35:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3749999999999992e-06 2024-04-24T14:35:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984246+0000", 2024-04-24T14:35:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2962999999999998e-05 2024-04-24T14:35:09.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:09.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984473+0000", 2024-04-24T14:35:09.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00022696399999999999 2024-04-24T14:35:09.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:09.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984771+0000", 2024-04-24T14:35:09.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00029739499999999999 2024-04-24T14:35:09.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:09.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992998+0000", 2024-04-24T14:35:09.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082275750000000009 2024-04-24T14:35:09.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:09.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994299+0000", 2024-04-24T14:35:09.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013005289999999999 2024-04-24T14:35:09.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:09.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994403+0000", 2024-04-24T14:35:09.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010376499999999999 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994482+0000", 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9013999999999999e-05 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:128 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.928636302999998, 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011784144, 2024-04-24T14:35:09.143 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:09.144 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:09.144 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:09.144 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:35:09.144 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:09.144 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 128 2024-04-24T14:35:09.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.144 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:09.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:09.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:35:09.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:09.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:35:09.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:09.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005324+0000", 2024-04-24T14:35:09.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:09.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:09.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005332+0000", 2024-04-24T14:35:09.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2989999999999997e-06 2024-04-24T14:35:09.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:09.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005354+0000", 2024-04-24T14:35:09.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1705999999999999e-05 2024-04-24T14:35:09.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:09.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005362+0000", 2024-04-24T14:35:09.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5599999999999996e-06 2024-04-24T14:35:09.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:09.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005418+0000", 2024-04-24T14:35:09.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6685999999999999e-05 2024-04-24T14:35:09.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:09.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005535+0000", 2024-04-24T14:35:09.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117008 2024-04-24T14:35:09.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:09.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005732+0000", 2024-04-24T14:35:09.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019718700000000001 2024-04-24T14:35:09.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:09.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016440+0000", 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010707787999999999 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016913+0000", 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047270400000000001 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017047+0000", 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013434699999999999 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:09.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017109+0000", 2024-04-24T14:35:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2192000000000004e-05 2024-04-24T14:35:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:130 3.1 e87/74 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 87'269, mlcod=87'265)", 2024-04-24T14:35:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:35:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 41.928183543000003, 2024-04-24T14:35:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013945631999999999, 2024-04-24T14:35:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:09.149 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:35:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:35:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005777+0000", 2024-04-24T14:35:09.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005791+0000", 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3743999999999999e-05 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005803+0000", 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2330999999999999e-05 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005810+0000", 2024-04-24T14:35:09.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8990000000000004e-06 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005888+0000", 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7872000000000003e-05 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005911+0000", 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3156e-05 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019642+0000", 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013730456 2024-04-24T14:35:09.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019724+0000", 2024-04-24T14:35:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1981e-05 2024-04-24T14:35:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:09.153 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:09.153 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:09.156 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:09.157 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:35:09.157 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:09.156+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:09.157 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:35:09.157 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:35:09.158 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-04-24T14:35:08.336359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":227157789,"num_objects":122,"num_object_clones":19,"num_object_copies":244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":21,"num_read":6319,"num_read_kb":177557,"num_write":956,"num_write_kb":273263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":445826026,"num_keys_recovered":150,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":969,"ondisk_log_size":969,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":38,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":1945308,"kb_used_data":438020,"kb_used_omap":8,"kb_used_meta":1507191,"kb_avail":469913892,"statfs":{"total":483183820800,"available":481191825408,"internally_reserved":0,"allocated":448532480,"data_stored":461432499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8520,"internal_metadata":1543364280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":624,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":501,"num_read_kb":2123,"num_write":22,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":54591561,"num_keys_recovered":27,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.120829"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":116,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.319464+0000","last_change":"2024-04-24T14:35:04.316837+0000","last_active":"2024-04-24T14:35:04.319464+0000","last_peered":"2024-04-24T14:35:04.319464+0000","last_clean":"2024-04-24T14:35:04.319464+0000","last_became_active":"2024-04-24T14:35:04.316535+0000","last_became_peered":"2024-04-24T14:35:04.316535+0000","last_unstale":"2024-04-24T14:35:04.319464+0000","last_undegraded":"2024-04-24T14:35:04.319464+0000","last_fullsized":"2024-04-24T14:35:04.319464+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:51:46.418293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062826099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":253,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.308540+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:35:04.308540+0000","last_peered":"2024-04-24T14:35:04.308540+0000","last_clean":"2024-04-24T14:35:04.308540+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:35:04.308540+0000","last_undegraded":"2024-04-24T14:35:04.308540+0000","last_fullsized":"2024-04-24T14:35:04.308540+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"49'3","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.392223+0000","last_change":"2024-04-24T14:35:04.391588+0000","last_active":"2024-04-24T14:35:04.392223+0000","last_peered":"2024-04-24T14:35:04.392223+0000","last_clean":"2024-04-24T14:35:04.392223+0000","last_became_active":"2024-04-24T14:35:04.316770+0000","last_became_peered":"2024-04-24T14:35:04.316770+0000","last_unstale":"2024-04-24T14:35:04.392223+0000","last_undegraded":"2024-04-24T14:35:04.392223+0000","last_fullsized":"2024-04-24T14:35:04.392223+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:06.338365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014822850000000001,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"109'115","reported_seq":203,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T14:35:05.873059+0000","last_change":"2024-04-24T14:35:04.690259+0000","last_active":"2024-04-24T14:35:05.873059+0000","last_peered":"2024-04-24T14:35:05.873059+0000","last_clean":"2024-04-24T14:35:05.873059+0000","last_became_active":"2024-04-24T14:35:04.319930+0000","last_became_peered":"2024-04-24T14:35:04.319930+0000","last_unstale":"2024-04-24T14:35:05.873059+0000","last_undegraded":"2024-04-24T14:35:05.873059+0000","last_fullsized":"2024-04-24T14:35:05.873059+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:52.537625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022812169999999999,"stat_sum":{"num_bytes":25165975,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":6,"num_read":1326,"num_read_kb":46882,"num_write":91,"num_write_kb":32846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":104988999,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"74'1","reported_seq":93,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.314206+0000","last_change":"2024-04-24T14:34:42.212529+0000","last_active":"2024-04-24T14:35:04.314206+0000","last_peered":"2024-04-24T14:35:04.314206+0000","last_clean":"2024-04-24T14:35:04.314206+0000","last_became_active":"2024-04-24T14:34:42.205009+0000","last_became_peered":"2024-04-24T14:34:42.205009+0000","last_unstale":"2024-04-24T14:35:04.314206+0000","last_undegraded":"2024-04-24T14:35:04.314206+0000","last_fullsized":"2024-04-24T14:35:04.314206+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:53.534797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039681800000000001,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"108'4","reported_seq":151,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.420280+0000","last_change":"2024-04-24T14:34:42.281331+0000","last_active":"2024-04-24T14:35:04.420280+0000","last_peered":"2024-04-24T14:35:04.420280+0000","last_clean":"2024-04-24T14:35:04.420280+0000","last_became_active":"2024-04-24T14:34:42.206964+0000","last_became_peered":"2024-04-24T14:34:42.206964+0000","last_unstale":"2024-04-24T14:35:04.420280+0000","last_undegraded":"2024-04-24T14:35:04.420280+0000","last_fullsized":"2024-04-24T14:35:04.420280+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:59:11.424740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079506599999999998,"stat_sum":{"num_bytes":12582990,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":12289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"108'353","reported_seq":2484,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T14:35:05.863025+0000","last_change":"2024-04-24T14:34:36.139031+0000","last_active":"2024-04-24T14:35:05.863025+0000","last_peered":"2024-04-24T14:35:05.863025+0000","last_clean":"2024-04-24T14:35:05.863025+0000","last_became_active":"2024-04-24T14:34:36.137865+0000","last_became_peered":"2024-04-24T14:34:36.137865+0000","last_unstale":"2024-04-24T14:35:05.863025+0000","last_undegraded":"2024-04-24T14:35:05.863025+0000","last_fullsized":"2024-04-24T14:35:05.863025+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:14.236340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079543300000000001,"stat_sum":{"num_bytes":31482010,"num_objects":20,"num_object_clones":9,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":6,"num_read":3365,"num_read_kb":74606,"num_write":240,"num_write_kb":56428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":25165862,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.318341+0000","last_change":"2024-04-24T14:35:04.316682+0000","last_active":"2024-04-24T14:35:04.318341+0000","last_peered":"2024-04-24T14:35:04.318341+0000","last_clean":"2024-04-24T14:35:04.318341+0000","last_became_active":"2024-04-24T14:35:04.315988+0000","last_became_peered":"2024-04-24T14:35:04.315988+0000","last_unstale":"2024-04-24T14:35:04.318341+0000","last_undegraded":"2024-04-24T14:35:04.318341+0000","last_fullsized":"2024-04-24T14:35:04.318341+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:51:37.583290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047250999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":162,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.315392+0000","last_change":"2024-04-24T14:34:42.329752+0000","last_active":"2024-04-24T14:35:04.315392+0000","last_peered":"2024-04-24T14:35:04.315392+0000","last_clean":"2024-04-24T14:35:04.315392+0000","last_became_active":"2024-04-24T14:34:42.207422+0000","last_became_peered":"2024-04-24T14:34:42.207422+0000","last_unstale":"2024-04-24T14:35:04.315392+0000","last_undegraded":"2024-04-24T14:35:04.315392+0000","last_fullsized":"2024-04-24T14:35:04.315392+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:02.081366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000475358,"stat_sum":{"num_bytes":8388684,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":5,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"108'238","reported_seq":690,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T14:35:05.863633+0000","last_change":"2024-04-24T14:34:44.842787+0000","last_active":"2024-04-24T14:35:05.863633+0000","last_peered":"2024-04-24T14:35:05.863633+0000","last_clean":"2024-04-24T14:35:05.863633+0000","last_became_active":"2024-04-24T14:34:44.841882+0000","last_became_peered":"2024-04-24T14:34:44.841882+0000","last_unstale":"2024-04-24T14:35:05.863633+0000","last_undegraded":"2024-04-24T14:35:05.863633+0000","last_fullsized":"2024-04-24T14:35:05.863633+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:47:40.933472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044041099999999998,"stat_sum":{"num_bytes":23118018,"num_objects":16,"num_object_clones":4,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":5,"num_read":959,"num_read_kb":38693,"num_write":325,"num_write_kb":34052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":43040844,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"74'1","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.315711+0000","last_change":"2024-04-24T14:34:42.334431+0000","last_active":"2024-04-24T14:35:04.315711+0000","last_peered":"2024-04-24T14:35:04.315711+0000","last_clean":"2024-04-24T14:35:04.315711+0000","last_became_active":"2024-04-24T14:34:42.207515+0000","last_became_peered":"2024-04-24T14:34:42.207515+0000","last_unstale":"2024-04-24T14:35:04.315711+0000","last_undegraded":"2024-04-24T14:35:04.315711+0000","last_fullsized":"2024-04-24T14:35:04.315711+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:16:14.137756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032986300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"108'7","reported_seq":27,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.536117+0000","last_change":"2024-04-24T14:35:04.535332+0000","last_active":"2024-04-24T14:35:04.536117+0000","last_peered":"2024-04-24T14:35:04.536117+0000","last_clean":"2024-04-24T14:35:04.536117+0000","last_became_active":"2024-04-24T14:35:04.318211+0000","last_became_peered":"2024-04-24T14:35:04.318211+0000","last_unstale":"2024-04-24T14:35:04.536117+0000","last_undegraded":"2024-04-24T14:35:04.536117+0000","last_fullsized":"2024-04-24T14:35:04.536117+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:53:27.959804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086195800000000002,"stat_sum":{"num_bytes":20971598,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":13,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":83886080,"num_keys_recovered":14,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"87'89","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.313544+0000","last_change":"2024-04-24T14:34:06.971354+0000","last_active":"2024-04-24T14:35:04.313544+0000","last_peered":"2024-04-24T14:35:04.313544+0000","last_clean":"2024-04-24T14:35:04.313544+0000","last_became_active":"2024-04-24T14:34:06.852598+0000","last_became_peered":"2024-04-24T14:34:06.852598+0000","last_unstale":"2024-04-24T14:35:04.313544+0000","last_undegraded":"2024-04-24T14:35:04.313544+0000","last_fullsized":"2024-04-24T14:35:04.313544+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:25:57.592466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052296799999999996,"stat_sum":{"num_bytes":12583042,"num_objects":12,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":4,"num_read":463,"num_read_kb":16975,"num_write":80,"num_write_kb":16420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777362,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"53'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.395796+0000","last_change":"2024-04-24T14:35:04.394903+0000","last_active":"2024-04-24T14:35:04.395796+0000","last_peered":"2024-04-24T14:35:04.395796+0000","last_clean":"2024-04-24T14:35:04.395796+0000","last_became_active":"2024-04-24T14:35:04.318139+0000","last_became_peered":"2024-04-24T14:35:04.318139+0000","last_unstale":"2024-04-24T14:35:04.395796+0000","last_undegraded":"2024-04-24T14:35:04.395796+0000","last_fullsized":"2024-04-24T14:35:04.395796+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:37.590793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054416299999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"108'13","reported_seq":50,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.387519+0000","last_change":"2024-04-24T14:34:42.557978+0000","last_active":"2024-04-24T14:35:04.387519+0000","last_peered":"2024-04-24T14:35:04.387519+0000","last_clean":"2024-04-24T14:35:04.387519+0000","last_became_active":"2024-04-24T14:34:42.208181+0000","last_became_peered":"2024-04-24T14:34:42.208181+0000","last_unstale":"2024-04-24T14:35:04.387519+0000","last_undegraded":"2024-04-24T14:35:04.387519+0000","last_fullsized":"2024-04-24T14:35:04.387519+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:32:37.282127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748852,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":16,"num_read_kb":135,"num_write":13,"num_write_kb":36868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":37748774,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.318437+0000","last_change":"2024-04-24T14:35:04.317455+0000","last_active":"2024-04-24T14:35:04.318437+0000","last_peered":"2024-04-24T14:35:04.318437+0000","last_clean":"2024-04-24T14:35:04.318437+0000","last_became_active":"2024-04-24T14:35:04.317129+0000","last_became_peered":"2024-04-24T14:35:04.317129+0000","last_unstale":"2024-04-24T14:35:04.318437+0000","last_undegraded":"2024-04-24T14:35:04.318437+0000","last_fullsized":"2024-04-24T14:35:04.318437+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:24:39.250920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030291500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"108'6","reported_seq":124,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.366366+0000","last_change":"2024-04-24T14:33:34.702742+0000","last_active":"2024-04-24T14:35:04.366366+0000","last_peered":"2024-04-24T14:35:04.366366+0000","last_clean":"2024-04-24T14:35:04.366366+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:35:04.366366+0000","last_undegraded":"2024-04-24T14:35:04.366366+0000","last_fullsized":"2024-04-24T14:35:04.366366+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064337400000000005,"stat_sum":{"num_bytes":20971598,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":128,"num_write":6,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'1","reported_seq":79,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.313059+0000","last_change":"2024-04-24T14:34:36.137425+0000","last_active":"2024-04-24T14:35:04.313059+0000","last_peered":"2024-04-24T14:35:04.313059+0000","last_clean":"2024-04-24T14:35:04.313059+0000","last_became_active":"2024-04-24T14:34:36.136250+0000","last_became_peered":"2024-04-24T14:34:36.136250+0000","last_unstale":"2024-04-24T14:35:04.313059+0000","last_undegraded":"2024-04-24T14:35:04.313059+0000","last_fullsized":"2024-04-24T14:35:04.313059+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:32:13.633169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013794250000000001,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"76'5","reported_seq":95,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.308953+0000","last_change":"2024-04-24T14:34:42.438578+0000","last_active":"2024-04-24T14:35:04.308953+0000","last_peered":"2024-04-24T14:35:04.308953+0000","last_clean":"2024-04-24T14:35:04.308953+0000","last_became_active":"2024-04-24T14:34:42.208652+0000","last_became_peered":"2024-04-24T14:34:42.208652+0000","last_unstale":"2024-04-24T14:35:04.308953+0000","last_undegraded":"2024-04-24T14:35:04.308953+0000","last_fullsized":"2024-04-24T14:35:04.308953+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:13:41.703506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059660799999999999,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165862,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'3","reported_seq":102,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.323711+0000","last_change":"2024-04-24T14:35:04.323260+0000","last_active":"2024-04-24T14:35:04.323711+0000","last_peered":"2024-04-24T14:35:04.323711+0000","last_clean":"2024-04-24T14:35:04.323711+0000","last_became_active":"2024-04-24T14:35:04.319983+0000","last_became_peered":"2024-04-24T14:35:04.319983+0000","last_unstale":"2024-04-24T14:35:04.323711+0000","last_undegraded":"2024-04-24T14:35:04.323711+0000","last_fullsized":"2024-04-24T14:35:04.323711+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:08.239076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000856414,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"84'6","reported_seq":159,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-24T14:35:04.308826+0000","last_change":"2024-04-24T14:34:06.924942+0000","last_active":"2024-04-24T14:35:04.308826+0000","last_peered":"2024-04-24T14:35:04.308826+0000","last_clean":"2024-04-24T14:35:04.308826+0000","last_became_active":"2024-04-24T14:34:06.852864+0000","last_became_peered":"2024-04-24T14:34:06.852864+0000","last_unstale":"2024-04-24T14:35:04.308826+0000","last_undegraded":"2024-04-24T14:35:04.308826+0000","last_fullsized":"2024-04-24T14:35:04.308826+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:42.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023873739999999998,"stat_sum":{"num_bytes":8388722,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":6,"num_write_kb":8196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":111,"num_keys_recovered":26,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":92349045,"num_objects":66,"num_object_clones":19,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":21,"num_read":6113,"num_read_kb":177156,"num_write":736,"num_write_kb":139746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":189973067,"num_keys_recovered":73,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194441216,"data_stored":211158288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":795,"ondisk_log_size":795,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":134218306,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":52,"num_read_kb":271,"num_write":55,"num_write_kb":131095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":255852848,"num_keys_recovered":51,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":373399552,"data_stored":373294364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49,"ondisk_log_size":49,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640294,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356056,"kb_used_data":54596,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94015784,"statfs":{"total":96636764160,"available":96272162816,"internally_reserved":0,"allocated":55906304,"data_stored":55139065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.80200000000000005,"15min":0.81499999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0720000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.80300000000000005,"15min":0.80900000000000005},"min":{"1min":0.42699999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.012,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.90400000000000003}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":1.022}]},{"osd":2,"last update":"Wed Apr 24 14:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.82499999999999996,"15min":0.84099999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.95799999999999996,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.751,"5min":0.84199999999999997,"15min":0.85699999999999998},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.98599999999999999,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.65300000000000002}]},{"osd":3,"last update":"Wed Apr 24 14:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.80500000000000005,"15min":0.82499999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.93899999999999995,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.874},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.84499999999999997,"15min":0.86699999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0349999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.76900000000000002}]},{"osd":4,"last update":"Wed Apr 24 14:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.872,"15min":0.876},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.5030000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.96399999999999997,"15min":0.98799999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.083,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.61499999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640294,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":423688,"kb_used_data":122228,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93948152,"statfs":{"total":96636764160,"available":96202907648,"internally_reserved":0,"allocated":125161472,"data_stored":132769883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.78400000000000003,"15min":0.78100000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.91000000000000003,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.79200000000000004}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.53500000000000003}]},{"osd":2,"last update":"Wed Apr 24 14:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.84699999999999998,"15min":0.86499999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.95199999999999996,"5min":1.113,"15min":1.113},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.84699999999999998,"15min":0.85499999999999998},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.161,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.90400000000000003}]},{"osd":3,"last update":"Wed Apr 24 14:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.84299999999999997,"15min":0.85499999999999998},"min":{"1min":0.56899999999999995,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.121,"5min":1.931,"15min":1.931},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.85999999999999999,"15min":0.874},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.393,"5min":1.591,"15min":1.591},"last":1.0089999999999999}]},{"osd":5,"last update":"Wed Apr 24 14:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.77400000000000002,"15min":0.78000000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.069,"5min":1.087,"15min":1.087},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.91100000000000003,"15min":0.92800000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1890000000000001,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.95699999999999996}]}]},{"osd":3,"up_from":11,"seq":47244640292,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377112,"kb_used_data":75660,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93994728,"statfs":{"total":96636764160,"available":96250601472,"internally_reserved":0,"allocated":77475840,"data_stored":76695846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.75900000000000001,"15min":0.77500000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.88500000000000001,"5min":1.282,"15min":1.282},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80200000000000005,"15min":0.80100000000000005},"min":{"1min":0.59299999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.131,"5min":1.208,"15min":1.208},"last":0.746}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.71599999999999997}]},{"osd":2,"last update":"Wed Apr 24 14:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.86099999999999999,"15min":0.875},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.087,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.88,"5min":0.93200000000000005,"15min":0.93999999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.268,"5min":1.337,"15min":1.337},"last":0.98399999999999999}]},{"osd":4,"last update":"Wed Apr 24 14:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.71699999999999997,"15min":0.72199999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.93100000000000005,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.78500000000000003,"15min":0.79400000000000004},"min":{"1min":0.58499999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.92800000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.54500000000000004}]},{"osd":5,"last update":"Wed Apr 24 14:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78800000000000003,"15min":0.78600000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.022,"5min":1.367,"15min":1.367},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.86199999999999999,"15min":0.86899999999999999},"min":{"1min":0.52800000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.169,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.61199999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640294,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344776,"kb_used_data":43316,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94027064,"statfs":{"total":96636764160,"available":96283713536,"internally_reserved":0,"allocated":44355584,"data_stored":43588383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.59799999999999998,"15min":0.60899999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.873,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.501}]},{"osd":1,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.56399999999999995}]},{"osd":3,"last update":"Wed Apr 24 14:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.84199999999999997,"15min":0.85399999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.014,"5min":1.147,"15min":1.147},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.89200000000000002,"15min":0.91100000000000003},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.111,"5min":1.77,"15min":1.77},"last":0.88900000000000001}]},{"osd":4,"last update":"Wed Apr 24 14:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.84699999999999998,"15min":0.85399999999999998},"min":{"1min":0.61299999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.0620000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":1.3420000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.874,"15min":0.89200000000000002},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.995,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.0840000000000001}]},{"osd":5,"last update":"Wed Apr 24 14:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.93700000000000006,"15min":0.96099999999999997},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.113,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.123},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.90000000000000002,"15min":0.92000000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.097,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":1.194}]}]},{"osd":1,"up_from":0,"seq":283467841550,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":11,"seq":47244640293,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443676,"kb_used_data":142220,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93928164,"statfs":{"total":96636764160,"available":96182439936,"internally_reserved":0,"allocated":145633280,"data_stored":153239322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 14:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.73699999999999999}]},{"osd":2,"last update":"Wed Apr 24 14:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58899999999999997,"15min":0.59499999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":1.0880000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67600000000000005,"15min":0.67800000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.83899999999999997}]},{"osd":3,"last update":"Wed Apr 24 14:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79800000000000004,"15min":0.80000000000000004},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80200000000000005,"15min":0.80400000000000005},"min":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.496,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":1.0469999999999999}]},{"osd":4,"last update":"Wed Apr 24 14:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.91400000000000003,"15min":0.92400000000000004},"min":{"1min":0.62,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.91500000000000004,"15min":0.91800000000000004},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3919999999999999,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.2110000000000001}]},{"osd":5,"last update":"Wed Apr 24 14:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.88,"15min":0.88700000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":1.256},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.92100000000000004,"15min":0.92700000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":1.1359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100691968,"data_stored":100663722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104873984,"data_stored":104857752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20979712,"data_stored":20971596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":58744832,"data_stored":58720524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33558528,"data_stored":33554510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43048960,"data_stored":51437852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22073344,"data_stored":22069286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":22081536,"data_stored":22069442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":21053440,"data_stored":21037207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65130496,"data_stored":73507294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":21053440,"data_stored":21037207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":51,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:35:09.160 INFO:tasks.thrashosds.thrasher:Setting 4.3 to [4, 1, 0, 5] 2024-04-24T14:35:09.160 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.3', '4', '1', '0', '5'] 2024-04-24T14:35:09.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 4.3 4 1 0 5 2024-04-24T14:35:09.162 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:35:09.162 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:35:09.162 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:35:09.162 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:35:09.162 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:35:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:35:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:35:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:35:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:35:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:35:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:35:09.255 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:09.258+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:09.270 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:09.270 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:09.280 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:35:09.302 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd16809 2024-04-24T14:35:09.302 INFO:tasks.workunit.client.0.smithi173.stderr:++ awk '/entries inspected/ {print $1}' 2024-04-24T14:35:09.356 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:09.358+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:09.378 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:09.379 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:09.388 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:35:09.395 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:35:09.400+0000 7f17dffff700 -1 JournalMetadata: 0x560c430ccd60 failed to locate client: INSPECT 2024-04-24T14:35:09.404 INFO:tasks.workunit.client.0.smithi173.stderr:+ queue_length=0 2024-04-24T14:35:09.404 INFO:tasks.workunit.client.0.smithi173.stderr:+ tries=4 2024-04-24T14:35:09.404 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 4 -lt 10 ']' 2024-04-24T14:35:09.404 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 0 -gt 0 ']' 2024-04-24T14:35:09.405 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T14:35:09.405 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device /dev/nbd1 19651 2024-04-24T14:35:09.405 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=/dev/nbd1 2024-04-24T14:35:09.405 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=19651 2024-04-24T14:35:09.405 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T14:35:09.405 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:35:09.405 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:35:09.406 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:35:09.407 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:35:09.409 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:35:09.409 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:35:09.409 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T14:35:09.457 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:09.460+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:09.483 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:09.483 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:09.492 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:35:09.530 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:35:09.530 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^19651\b' 2024-04-24T14:35:09.530 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^19651\b' 2024-04-24T14:35:09.558 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:09.560+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:09.592 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:09.592 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:09.593 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:35:09.594 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:35:09.594 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *19651\b' 2024-04-24T14:35:09.594 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *19651\b' 2024-04-24T14:35:09.601 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:35:09.605 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:35:09.605 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:35:09.659 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:09.662+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:09.694 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:09.695 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:09.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:35:09.760 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:09.762+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:09.799 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:09.799 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:09.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:35:09.861 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:09.862+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:09.950 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:09.950 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:09.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:35:09.962 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:09.964+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:10.056 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:10.056 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:10.063 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:10.066+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:10.065 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:35:10.106 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:35:10.107 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd feature disable rbd/testrbdnbd16809 journaling 2024-04-24T14:35:10.163 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:10.168+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:10.214 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:10.214 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:10.221 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd config image rm rbd/testrbdnbd16809 rbd_discard_granularity_bytes 2024-04-24T14:35:10.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:35:10.264 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:10.268+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:10.328 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:35:10.328 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:35:10.328 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:35:10.329 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:10.329 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:10.330 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:35:10.330 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:35:10.333 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:35:10.333 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:35:10.333 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:35:10.341 INFO:teuthology.orchestra.run.smithi090.stderr:set 4.3 pg_upmap_items mapping to [4->1,0->5] 2024-04-24T14:35:10.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:35:10.365 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:10.368+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:10.445 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:10.445 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:10.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:35:10.460 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T14:35:10.466 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:10.468+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:10.556 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:10.556 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:10.565 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:35:10.567 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:10.570+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:10.667 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:10.672+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:10.675 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:10.676 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:10.685 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:35:10.768 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:10.770+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:10.786 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd1 2024-04-24T14:35:10.787 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:35:10.787 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:35:10.787 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:35:10.787 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:35:10.787 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd1'\'']/id' 2024-04-24T14:35:10.788 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:10.788 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:10.798 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:35:10.869 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:10.872+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:10.879 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=20093 2024-04-24T14:35:10.879 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 20093 2024-04-24T14:35:10.879 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 20093 -C rbd-nbd 2024-04-24T14:35:10.888 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:35:10.889 INFO:tasks.workunit.client.0.smithi173.stdout: 20093 ? 00:00:00 rbd-nbd 2024-04-24T14:35:10.889 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd feature enable rbd/testrbdnbd16809 journaling 2024-04-24T14:35:10.893 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:10.893 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:10.902 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:35:10.970 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:10.972+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:10.991 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-04-24T14:35:10.991 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:35:10.991 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:35:10.994 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:35:10.994 INFO:tasks.workunit.client.0.smithi173.stderr:++ which blkdiscard 2024-04-24T14:35:10.996 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/sbin/blkdiscard 2024-04-24T14:35:10.996 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:35:10.997 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd1 2024-04-24T14:35:11.010 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:11.011 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:11.020 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:35:11.032 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd feature disable rbd/testrbdnbd16809 journaling 2024-04-24T14:35:11.071 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:11.072+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:11.138 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device /dev/nbd1 20093 2024-04-24T14:35:11.139 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=/dev/nbd1 2024-04-24T14:35:11.139 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=20093 2024-04-24T14:35:11.139 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T14:35:11.139 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:35:11.139 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:35:11.140 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:35:11.141 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:35:11.144 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:35:11.144 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:35:11.144 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T14:35:11.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:35:11.170 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:11.170 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:11.179 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:35:11.210 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:11.214+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:11.253 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:35:11.253 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^20093\b' 2024-04-24T14:35:11.253 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^20093\b' 2024-04-24T14:35:11.271 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:11.274+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:11.275 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:11.275 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:11.284 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:35:11.309 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:35:11.317 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:35:11.318 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:35:11.319 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *20093\b' 2024-04-24T14:35:11.319 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *20093\b' 2024-04-24T14:35:11.328 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:35:11.329 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:35:11.331 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:35:11.352 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-24T14:35:11.353 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:35:11.372 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:11.374+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:11.374 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:35:11.387 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:11.387 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:11.394 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:35:11.396 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:35:11.419 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:35:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:35:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:35:11.422 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:35:11.423 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:35:11.423 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:35:11.478 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:11.482+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:11.492 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:11.493 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:11.493 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:35:11.493 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:35:11.493 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:11.502 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:35:11.574 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:11.578+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:11.599 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:11.599 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:35:11.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:35:11.600 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:35:11.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.600 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:35:11.600 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:11.600 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 91.530802993999998, 2024-04-24T14:35:11.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:35:11.600 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.600 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:11.600 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:11.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:11.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:35:11.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:11.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:35:11.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:35:11.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:35:11.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:35:11.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:35:11.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:35:11.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:35:11.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:35:11.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:35:11.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:35:11.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:11.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.535988415000006, 2024-04-24T14:35:11.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:35:11.605 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.605 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:11.605 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:11.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:11.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:35:11.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:11.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:35:11.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:35:11.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:35:11.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:35:11.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:35:11.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:35:11.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:35:11.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:35:11.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.608 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.609 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:35:11.609 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:11.609 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.525059157000001, 2024-04-24T14:35:11.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:35:11.609 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.609 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:11.609 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:11.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:11.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:35:11.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:11.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:35:11.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:35:11.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:35:11.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:35:11.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:11.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:35:11.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:35:11.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:35:11.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:35:11.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.613 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.613 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:35:11.613 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:11.613 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.514103075999998, 2024-04-24T14:35:11.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:35:11.613 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.613 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:11.614 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:11.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:11.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:35:11.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:11.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:35:11.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:35:11.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:35:11.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:35:11.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:35:11.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:35:11.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:35:11.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:35:11.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:35:11.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:35:11.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:11.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:35:11.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:35:11.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.617 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.617 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.617 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:35:11.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:35:11.617 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.617 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.617 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.618 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:35:11.618 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:11.618 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.502533710999998, 2024-04-24T14:35:11.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:35:11.618 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.618 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:11.618 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.618 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.618 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.618 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:11.618 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:35:11.619 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:35:11.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:35:11.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:35:11.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:35:11.620 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:35:11.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.620 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.620 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.620 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:35:11.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:35:11.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:35:11.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:35:11.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:11.621 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:35:11.621 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:35:11.621 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.621 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.621 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.622 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:35:11.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:35:11.622 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.622 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.622 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.622 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.622 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:35:11.622 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:11.622 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.249414052000006, 2024-04-24T14:35:11.622 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:35:11.622 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.622 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:11.623 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:11.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.623 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:11.623 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.623 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.623 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:35:11.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:11.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:35:11.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:35:11.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.624 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.624 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:35:11.624 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:35:11.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:35:11.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:35:11.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:35:11.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:35:11.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.625 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.625 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.625 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:35:11.625 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:35:11.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:11.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:35:11.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:35:11.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.626 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.626 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.626 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:35:11.626 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:35:11.626 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.626 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.626 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.627 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:35:11.627 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:11.627 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.238668884000006, 2024-04-24T14:35:11.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:35:11.627 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.627 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:11.627 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.627 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.627 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.627 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:11.627 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:11.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:35:11.628 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:11.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.628 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.628 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.628 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:35:11.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:35:11.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:35:11.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:35:11.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.629 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:35:11.629 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:35:11.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.629 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.629 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:35:11.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:35:11.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:35:11.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:35:11.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.630 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.630 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:11.630 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:35:11.630 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:35:11.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.631 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.631 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:35:11.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:35:11.631 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.631 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.631 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.631 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.631 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:35:11.631 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:11.631 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.226302607999997, 2024-04-24T14:35:11.631 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:35:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:35:11.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:11.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:35:11.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:35:11.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.633 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.633 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.633 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:35:11.633 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:35:11.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:35:11.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:35:11.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.634 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.634 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:35:11.634 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:35:11.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:35:11.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:35:11.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:11.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:35:11.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:35:11.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.635 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.635 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.635 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:35:11.635 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:35:11.636 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.636 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.636 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.636 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:35:11.636 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:11.636 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.072250800999996, 2024-04-24T14:35:11.636 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:35:11.636 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.636 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:11.636 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.636 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:11.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:11.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.637 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.637 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:35:11.637 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:11.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:35:11.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:35:11.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:35:11.638 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:35:11.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.638 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.638 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.638 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:35:11.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:35:11.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:35:11.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:35:11.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.639 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:35:11.639 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:35:11.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.639 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.639 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:11.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:35:11.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:35:11.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.640 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.640 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:35:11.640 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:35:11.640 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.640 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.640 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.640 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.640 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:35:11.641 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:11.641 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 76.902412124999998, 2024-04-24T14:35:11.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:35:11.641 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.641 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:11.641 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:11.641 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.641 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.641 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.641 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:11.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:35:11.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:11.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.642 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.642 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:35:11.642 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:35:11.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.642 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:35:11.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:35:11.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:35:11.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:35:11.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.643 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.643 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.643 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:35:11.643 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:35:11.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:35:11.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:35:11.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:11.644 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:35:11.644 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:35:11.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.644 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.644 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.645 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:35:11.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:35:11.645 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.645 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.645 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.645 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.645 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:35:11.645 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:11.645 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 72.567441290000005, 2024-04-24T14:35:11.645 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:35:11.645 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.645 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:11.645 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:11.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.646 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:11.646 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.646 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.646 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.646 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:11.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:11.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:35:11.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:35:11.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.647 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.647 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:35:11.647 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:35:11.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.647 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:35:11.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:35:11.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:35:11.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:35:11.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.648 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.648 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:35:11.648 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:35:11.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:11.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:35:11.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:35:11.649 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.649 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.649 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.649 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:35:11.649 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:35:11.649 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.649 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.649 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.650 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:35:11.650 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:11.650 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 68.723290542000001, 2024-04-24T14:35:11.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:35:11.650 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.650 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:35:11.650 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.650 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.650 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.650 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:11.650 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:11.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.651 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.651 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:11.651 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:11.651 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:35:11.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:35:11.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:35:11.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:35:11.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.652 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.652 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:35:11.652 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:35:11.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:35:11.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:35:11.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.653 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.653 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.653 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:35:11.653 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:35:11.653 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.653 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.653 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.653 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.654 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:35:11.654 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:11.654 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 53.499527542000003, 2024-04-24T14:35:11.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044084859999999997, 2024-04-24T14:35:11.654 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.654 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:11.654 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.654 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.654 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:11.654 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.654 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:11.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103206+0000", 2024-04-24T14:35:11.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:11.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.655 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.655 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.655 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104731+0000", 2024-04-24T14:35:11.655 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001524755 2024-04-24T14:35:11.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104759+0000", 2024-04-24T14:35:11.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7973000000000001e-05 2024-04-24T14:35:11.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.656 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104807+0000", 2024-04-24T14:35:11.656 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8112000000000003e-05 2024-04-24T14:35:11.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.656 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.656 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104907+0000", 2024-04-24T14:35:11.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100317 2024-04-24T14:35:11.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104918+0000", 2024-04-24T14:35:11.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747999999999999e-05 2024-04-24T14:35:11.657 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.657 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.657 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.657 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.107616+0000", 2024-04-24T14:35:11.657 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026977559999999999 2024-04-24T14:35:11.658 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.658 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.658 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.658 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.658 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:35:11.658 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:11.658 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 53.476232588999999, 2024-04-24T14:35:11.658 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046193859999999996, 2024-04-24T14:35:11.658 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.658 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:11.658 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:11.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:11.659 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.659 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.659 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.659 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126501+0000", 2024-04-24T14:35:11.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:11.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.660 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.660 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128169+0000", 2024-04-24T14:35:11.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016686229999999999 2024-04-24T14:35:11.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.660 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.660 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128202+0000", 2024-04-24T14:35:11.660 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2539000000000001e-05 2024-04-24T14:35:11.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128222+0000", 2024-04-24T14:35:11.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0149e-05 2024-04-24T14:35:11.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.661 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128375+0000", 2024-04-24T14:35:11.661 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000153459 2024-04-24T14:35:11.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.661 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.661 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128399+0000", 2024-04-24T14:35:11.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3393000000000001e-05 2024-04-24T14:35:11.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.662 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.662 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.131122+0000", 2024-04-24T14:35:11.662 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027227140000000002 2024-04-24T14:35:11.662 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.662 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.662 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.662 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.662 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5674.0:27 3.2 3:5592b7cf:::journal.12ff64619e2d:head [watch watch cookie 139908626846656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T14:35:11.663 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.218271+0000", 2024-04-24T14:35:11.663 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 7.384463996, 2024-04-24T14:35:11.663 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.107948372, 2024-04-24T14:35:11.663 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.663 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:11.663 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:11.663 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5674", 2024-04-24T14:35:11.663 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1498157813", 2024-04-24T14:35:11.663 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 27 2024-04-24T14:35:11.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.663 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.663 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.663 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218271+0000", 2024-04-24T14:35:11.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218271+0000", 2024-04-24T14:35:11.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.664 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.664 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.664 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218267+0000", 2024-04-24T14:35:11.664 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:35:11.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218280+0000", 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2469e-05 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218303+0000", 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2952999999999999e-05 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218316+0000", 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.313e-05 2024-04-24T14:35:11.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218446+0000", 2024-04-24T14:35:11.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000130154 2024-04-24T14:35:11.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:35:11.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218460+0000", 2024-04-24T14:35:11.666 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3852999999999999e-05 2024-04-24T14:35:11.666 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.666 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.666 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.666 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.320512+0000", 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.102051709 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for missing object", 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.320660+0000", 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000148362 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.322855+0000", 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021955249999999998 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:11.667 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.322976+0000", 2024-04-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000120813 2024-04-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.323702+0000", 2024-04-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00072575199999999997 2024-04-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.323827+0000", 2024-04-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012478200000000001 2024-04-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.668 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.324040+0000", 2024-04-24T14:35:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021278700000000001 2024-04-24T14:35:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.324936+0000", 2024-04-24T14:35:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00089639600000000004 2024-04-24T14:35:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.326067+0000", 2024-04-24T14:35:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011309110000000001 2024-04-24T14:35:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.669 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.326114+0000", 2024-04-24T14:35:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6903999999999997e-05 2024-04-24T14:35:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.326219+0000", 2024-04-24T14:35:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105368 2024-04-24T14:35:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.670 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 108/107 [PushOp(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 49'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@49'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:35:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.396664+0000", 2024-04-24T14:35:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 7.2060710779999999, 2024-04-24T14:35:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004962241, 2024-04-24T14:35:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.396664+0000", 2024-04-24T14:35:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.671 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.396664+0000", 2024-04-24T14:35:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.396662+0000", 2024-04-24T14:35:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.398018+0000", 2024-04-24T14:35:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001355904 2024-04-24T14:35:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.672 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.398037+0000", 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8907e-05 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.398049+0000", 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1661000000000001e-05 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.400590+0000", 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025413749999999998 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.673 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.400599+0000", 2024-04-24T14:35:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0420000000000008e-06 2024-04-24T14:35:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.401626+0000", 2024-04-24T14:35:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0010267119999999999 2024-04-24T14:35:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:8 2.2 2:4f9137c5:::journal_data.2.12ff64619e2d.3:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e108)", 2024-04-24T14:35:11.674 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.439647+0000", 2024-04-24T14:35:11.675 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 7.1630874379999998, 2024-04-24T14:35:11.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050746209999999996, 2024-04-24T14:35:11.675 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.675 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:11.675 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439647+0000", 2024-04-24T14:35:11.675 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.675 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.675 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.675 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439647+0000", 2024-04-24T14:35:11.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439646+0000", 2024-04-24T14:35:11.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:11.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.676 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439652+0000", 2024-04-24T14:35:11.676 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0170000000000004e-06 2024-04-24T14:35:11.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.676 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.676 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439660+0000", 2024-04-24T14:35:11.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7730000000000005e-06 2024-04-24T14:35:11.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439665+0000", 2024-04-24T14:35:11.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5220000000000001e-06 2024-04-24T14:35:11.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.677 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.677 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.441882+0000", 2024-04-24T14:35:11.677 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022169260000000001 2024-04-24T14:35:11.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:11.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.442038+0000", 2024-04-24T14:35:11.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000156661 2024-04-24T14:35:11.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.443062+0000", 2024-04-24T14:35:11.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001023604 2024-04-24T14:35:11.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.678 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.678 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.678 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.443281+0000", 2024-04-24T14:35:11.678 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000218564 2024-04-24T14:35:11.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.679 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.679 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.444722+0000", 2024-04-24T14:35:11.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0014413539999999999 2024-04-24T14:35:11.679 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.679 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.679 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.679 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.679 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5674.0:200 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 3895296~36864] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-04-24T14:35:11.679 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.470076+0000", 2024-04-24T14:35:11.679 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 7.1326590769999996, 2024-04-24T14:35:11.679 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044721860000000004, 2024-04-24T14:35:11.680 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.680 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:11.680 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:11.680 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5674", 2024-04-24T14:35:11.680 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1498157813", 2024-04-24T14:35:11.680 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 200 2024-04-24T14:35:11.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.680 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.680 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.680 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470076+0000", 2024-04-24T14:35:11.680 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470076+0000", 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470075+0000", 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470078+0000", 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5459999999999998e-06 2024-04-24T14:35:11.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470083+0000", 2024-04-24T14:35:11.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2009999999999998e-06 2024-04-24T14:35:11.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470088+0000", 2024-04-24T14:35:11.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7480000000000001e-06 2024-04-24T14:35:11.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.682 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.682 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.682 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473569+0000", 2024-04-24T14:35:11.682 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034816399999999998 2024-04-24T14:35:11.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.683 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473887+0000", 2024-04-24T14:35:11.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00031747200000000001 2024-04-24T14:35:11.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.683 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.683 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.683 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.474548+0000", 2024-04-24T14:35:11.683 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066107099999999995 2024-04-24T14:35:11.683 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.683 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.683 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.684 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 108/107 [PushOp(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 49'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@49'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:35:11.684 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.470456+0000", 2024-04-24T14:35:11.684 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 7.1322790439999997, 2024-04-24T14:35:11.684 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055876789999999999, 2024-04-24T14:35:11.684 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.684 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:11.684 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.684 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.684 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.684 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470456+0000", 2024-04-24T14:35:11.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470456+0000", 2024-04-24T14:35:11.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.685 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470454+0000", 2024-04-24T14:35:11.685 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:11.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.685 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.685 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470992+0000", 2024-04-24T14:35:11.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053743599999999997 2024-04-24T14:35:11.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.471024+0000", 2024-04-24T14:35:11.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2480000000000001e-05 2024-04-24T14:35:11.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.686 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.686 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.471045+0000", 2024-04-24T14:35:11.686 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0503e-05 2024-04-24T14:35:11.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.686 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473545+0000", 2024-04-24T14:35:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025000299999999999 2024-04-24T14:35:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473565+0000", 2024-04-24T14:35:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0352000000000001e-05 2024-04-24T14:35:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.476043+0000", 2024-04-24T14:35:11.687 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024783840000000001 2024-04-24T14:35:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5606.0:3 3.2 3:5592b7cf:::journal.12ff64619e2d:head [watch watch cookie 94231437332288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-24T14:35:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.527849+0000", 2024-04-24T14:35:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 7.0748854860000003, 2024-04-24T14:35:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044396410000000002, 2024-04-24T14:35:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5606", 2024-04-24T14:35:11.688 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2386300588", 2024-04-24T14:35:11.689 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 3 2024-04-24T14:35:11.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.689 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:11.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.689 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:11.689 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527849+0000", 2024-04-24T14:35:11.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.689 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:11.689 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527849+0000", 2024-04-24T14:35:11.689 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:11.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:11.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527848+0000", 2024-04-24T14:35:11.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:11.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:11.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527855+0000", 2024-04-24T14:35:11.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8719999999999996e-06 2024-04-24T14:35:11.690 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.690 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.690 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:11.690 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527890+0000", 2024-04-24T14:35:11.690 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4836999999999999e-05 2024-04-24T14:35:11.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:11.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527906+0000", 2024-04-24T14:35:11.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6535999999999999e-05 2024-04-24T14:35:11.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:11.691 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.529465+0000", 2024-04-24T14:35:11.691 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001559429 2024-04-24T14:35:11.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.691 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.691 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:11.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.529800+0000", 2024-04-24T14:35:11.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000334638 2024-04-24T14:35:11.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:11.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.530413+0000", 2024-04-24T14:35:11.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061266800000000002 2024-04-24T14:35:11.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.692 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:11.692 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532044+0000", 2024-04-24T14:35:11.692 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001631565 2024-04-24T14:35:11.692 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.692 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:11.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532061+0000", 2024-04-24T14:35:11.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6874000000000001e-05 2024-04-24T14:35:11.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:11.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532139+0000", 2024-04-24T14:35:11.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8191999999999999e-05 2024-04-24T14:35:11.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:11.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:11.693 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:11.693 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532289+0000", 2024-04-24T14:35:11.693 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014948199999999999 2024-04-24T14:35:11.694 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.694 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.694 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.694 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:11.694 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:11.694 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:11.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:35:11.696 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:11.702+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:11.796 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:11.800+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:11.807 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:11.807 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:35:11.807 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:11.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:35:11.830 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:35:11.831 INFO:tasks.workunit.client.0.smithi173.stderr:++ ceph-conf --show-config-value librados_thread_count 2024-04-24T14:35:11.854 INFO:tasks.workunit.client.0.smithi173.stderr:+ EXPECTED=2 2024-04-24T14:35:11.855 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:35:11.855 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:35:11.856 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:35:11.858 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:35:11.858 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:35:11.861 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:35:11.861 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:35:11.861 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd16809 2024-04-24T14:35:11.897 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:11.898+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:11.916 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:11.916 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:11.916 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:11.916 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:11.916 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:11.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:35:11.998 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:12.002+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:12.024 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.148353807999996, 2024-04-24T14:35:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:35:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:35:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:35:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:35:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:35:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:35:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:35:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:35:12.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:35:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:35:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:35:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:35:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:35:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.148324822999996, 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:35:12.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:35:12.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:35:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:35:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:35:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:35:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:35:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:12.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:35:12.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.148306337999998, 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:12.036 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:35:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:35:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:35:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:35:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:35:12.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:35:12.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:35:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:35:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:35:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:35:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:35:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.148291056999994, 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:35:12.041 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:35:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:35:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:35:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:35:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:35:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:35:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:35:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.046 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.148275276000007, 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:35:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:35:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:35:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:35:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:35:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:35:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.148260988000004, 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.052 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:35:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:35:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:35:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:35:12.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:35:12.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:35:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:35:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:35:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:35:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:35:12.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:35:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:35:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:35:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:35:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:35:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.148241236000004, 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:35:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:35:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:35:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:35:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:35:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:35:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.148223788999999, 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:35:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:35:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:35:12.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:35:12.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:35:12.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.068 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:100 3.2 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.274168973999998, 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009078015, 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 100 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754019+0000", 2024-04-24T14:35:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754025+0000", 2024-04-24T14:35:12.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2110000000000003e-06 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754037+0000", 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1868999999999999e-05 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754046+0000", 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7430000000000008e-06 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754095+0000", 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8924000000000001e-05 2024-04-24T14:35:12.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754153+0000", 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8165000000000003e-05 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754262+0000", 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108694 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.762999+0000", 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087372590000000007 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763027+0000", 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8467e-05 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763070+0000", 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2236e-05 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763098+0000", 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8561999999999999e-05 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.073 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:101 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [delete] snapc 7=[7,6,4] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.068961307999999, 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012894743, 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 101 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.074 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959227+0000", 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959233+0000", 2024-04-24T14:35:12.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.857e-06 2024-04-24T14:35:12.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959249+0000", 2024-04-24T14:35:12.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6155999999999999e-05 2024-04-24T14:35:12.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959257+0000", 2024-04-24T14:35:12.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7440000000000007e-06 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959790+0000", 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053381100000000003 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960206+0000", 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041549700000000002 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960744+0000", 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053762199999999997 2024-04-24T14:35:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971963+0000", 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011219816000000001 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972021+0000", 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7855000000000001e-05 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972073+0000", 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2196e-05 2024-04-24T14:35:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972123+0000", 2024-04-24T14:35:12.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9311999999999998e-05 2024-04-24T14:35:12.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.079 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.079 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:102 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.079 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:12.079 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.068921281999998, 2024-04-24T14:35:12.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012939713, 2024-04-24T14:35:12.079 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 102 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:12.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959267+0000", 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959274+0000", 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.593e-06 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959291+0000", 2024-04-24T14:35:12.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6387000000000001e-05 2024-04-24T14:35:12.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959301+0000", 2024-04-24T14:35:12.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0023e-05 2024-04-24T14:35:12.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969594+0000", 2024-04-24T14:35:12.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010293182 2024-04-24T14:35:12.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969902+0000", 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030835299999999998 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970408+0000", 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050588100000000002 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972009+0000", 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016011210000000001 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972132+0000", 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122521 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972174+0000", 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2242e-05 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972208+0000", 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3522999999999997e-05 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.085 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:104 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.085 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:12.085 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.068310068000002, 2024-04-24T14:35:12.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014589062, 2024-04-24T14:35:12.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.085 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.085 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 104 2024-04-24T14:35:12.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.085 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959877+0000", 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959890+0000", 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2537e-05 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959909+0000", 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9340000000000001e-05 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959922+0000", 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2882e-05 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970900+0000", 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010977736 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971039+0000", 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139593 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971453+0000", 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041351099999999998 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974044+0000", 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025908289999999998 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974377+0000", 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033310399999999997 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974433+0000", 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6166999999999999e-05 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974468+0000", 2024-04-24T14:35:12.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5336000000000002e-05 2024-04-24T14:35:12.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.090 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.090 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:106 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.090 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:12.090 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.068250734999999, 2024-04-24T14:35:12.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017192064999999999, 2024-04-24T14:35:12.090 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.090 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.090 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.090 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 106 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959938+0000", 2024-04-24T14:35:12.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959943+0000", 2024-04-24T14:35:12.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.677e-06 2024-04-24T14:35:12.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959953+0000", 2024-04-24T14:35:12.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8069999999999994e-06 2024-04-24T14:35:12.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0560000000000001e-06 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972232+0000", 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012272460000000001 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972304+0000", 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2630000000000001e-05 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972548+0000", 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000244047 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976900+0000", 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043514069999999998 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977076+0000", 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176281 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977106+0000", 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0015e-05 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977131+0000", 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4669e-05 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:114 3.2 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.095 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.044929570999997, 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090880089999999993, 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 114 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983259+0000", 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983264+0000", 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9130000000000002e-06 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983277+0000", 2024-04-24T14:35:12.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3723999999999999e-05 2024-04-24T14:35:12.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983282+0000", 2024-04-24T14:35:12.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0429999999999998e-06 2024-04-24T14:35:12.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983339+0000", 2024-04-24T14:35:12.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6323000000000003e-05 2024-04-24T14:35:12.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983464+0000", 2024-04-24T14:35:12.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012554299999999999 2024-04-24T14:35:12.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:12.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983693+0000", 2024-04-24T14:35:12.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000228848 2024-04-24T14:35:12.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.986978+0000", 2024-04-24T14:35:12.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032852810000000001 2024-04-24T14:35:12.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992288+0000", 2024-04-24T14:35:12.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053095369999999996 2024-04-24T14:35:12.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992322+0000", 2024-04-24T14:35:12.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3683999999999998e-05 2024-04-24T14:35:12.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992348+0000", 2024-04-24T14:35:12.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6157999999999999e-05 2024-04-24T14:35:12.101 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.101 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.101 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.101 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:121 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.101 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:12.101 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.032873471999999, 2024-04-24T14:35:12.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098798159999999996, 2024-04-24T14:35:12.101 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.102 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.102 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.102 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.102 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.102 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 121 2024-04-24T14:35:12.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.102 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:12.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:12.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995315+0000", 2024-04-24T14:35:12.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995320+0000", 2024-04-24T14:35:12.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0239999999999997e-06 2024-04-24T14:35:12.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995328+0000", 2024-04-24T14:35:12.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7800000000000001e-06 2024-04-24T14:35:12.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995334+0000", 2024-04-24T14:35:12.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2330000000000003e-06 2024-04-24T14:35:12.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998810+0000", 2024-04-24T14:35:12.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034756090000000002 2024-04-24T14:35:12.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998870+0000", 2024-04-24T14:35:12.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0625999999999999e-05 2024-04-24T14:35:12.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:12.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.999091+0000", 2024-04-24T14:35:12.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000220631 2024-04-24T14:35:12.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.001389+0000", 2024-04-24T14:35:12.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022977549999999998 2024-04-24T14:35:12.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005097+0000", 2024-04-24T14:35:12.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037085439999999998 2024-04-24T14:35:12.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005154+0000", 2024-04-24T14:35:12.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6208000000000001e-05 2024-04-24T14:35:12.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005196+0000", 2024-04-24T14:35:12.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2052000000000003e-05 2024-04-24T14:35:12.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.107 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.107 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:127 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.107 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:12.107 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.022571956, 2024-04-24T14:35:12.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013414918, 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:12.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005616+0000", 2024-04-24T14:35:12.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005622+0000", 2024-04-24T14:35:12.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2020000000000003e-06 2024-04-24T14:35:12.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005632+0000", 2024-04-24T14:35:12.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2024-04-24T14:35:12.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005637+0000", 2024-04-24T14:35:12.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7439999999999998e-06 2024-04-24T14:35:12.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005947+0000", 2024-04-24T14:35:12.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000310792 2024-04-24T14:35:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006059+0000", 2024-04-24T14:35:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011192399999999999 2024-04-24T14:35:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006503+0000", 2024-04-24T14:35:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000443394 2024-04-24T14:35:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018864+0000", 2024-04-24T14:35:12.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012361162 2024-04-24T14:35:12.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018877+0000", 2024-04-24T14:35:12.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3497e-05 2024-04-24T14:35:12.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018965+0000", 2024-04-24T14:35:12.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7476000000000001e-05 2024-04-24T14:35:12.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019032+0000", 2024-04-24T14:35:12.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7469000000000001e-05 2024-04-24T14:35:12.113 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.113 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.113 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.113 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:129 3.2 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.113 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:12.113 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.022431722999997, 2024-04-24T14:35:12.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010059754000000001, 2024-04-24T14:35:12.113 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.114 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.114 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 129 2024-04-24T14:35:12.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.114 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:12.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005757+0000", 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005760+0000", 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2430000000000001e-06 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005769+0000", 2024-04-24T14:35:12.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4810000000000006e-06 2024-04-24T14:35:12.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005773+0000", 2024-04-24T14:35:12.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:35:12.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005840+0000", 2024-04-24T14:35:12.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6934999999999995e-05 2024-04-24T14:35:12.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005961+0000", 2024-04-24T14:35:12.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012153 2024-04-24T14:35:12.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:12.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006181+0000", 2024-04-24T14:35:12.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021947500000000001 2024-04-24T14:35:12.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015520+0000", 2024-04-24T14:35:12.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093393049999999991 2024-04-24T14:35:12.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015733+0000", 2024-04-24T14:35:12.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021355699999999999 2024-04-24T14:35:12.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015773+0000", 2024-04-24T14:35:12.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9554000000000001e-05 2024-04-24T14:35:12.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015817+0000", 2024-04-24T14:35:12.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4325e-05 2024-04-24T14:35:12.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.120 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.120 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:131 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.120 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:12.121 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.022153842000002, 2024-04-24T14:35:12.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014251976, 2024-04-24T14:35:12.121 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.121 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.121 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131 2024-04-24T14:35:12.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.121 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:12.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:12.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006034+0000", 2024-04-24T14:35:12.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:12.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006042+0000", 2024-04-24T14:35:12.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2970000000000004e-06 2024-04-24T14:35:12.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006061+0000", 2024-04-24T14:35:12.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8505999999999999e-05 2024-04-24T14:35:12.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006071+0000", 2024-04-24T14:35:12.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0409e-05 2024-04-24T14:35:12.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007899+0000", 2024-04-24T14:35:12.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018274789999999999 2024-04-24T14:35:12.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008281+0000", 2024-04-24T14:35:12.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038244700000000001 2024-04-24T14:35:12.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:12.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008528+0000", 2024-04-24T14:35:12.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024723900000000001 2024-04-24T14:35:12.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019070+0000", 2024-04-24T14:35:12.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010541867999999999 2024-04-24T14:35:12.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020099+0000", 2024-04-24T14:35:12.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00102852 2024-04-24T14:35:12.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020226+0000", 2024-04-24T14:35:12.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012744300000000001 2024-04-24T14:35:12.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020287+0000", 2024-04-24T14:35:12.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1253000000000005e-05 2024-04-24T14:35:12.126 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.126 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.126 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.127 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:132 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.127 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:12.127 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.022099988999997, 2024-04-24T14:35:12.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015917981000000001, 2024-04-24T14:35:12.127 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.127 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.127 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.127 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.127 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.127 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 132 2024-04-24T14:35:12.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.127 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:12.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:12.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:12.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006093+0000", 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3819999999999997e-06 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006103+0000", 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0352e-05 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006110+0000", 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3910000000000003e-06 2024-04-24T14:35:12.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015586+0000", 2024-04-24T14:35:12.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094759119999999995 2024-04-24T14:35:12.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015730+0000", 2024-04-24T14:35:12.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014487199999999999 2024-04-24T14:35:12.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:12.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015886+0000", 2024-04-24T14:35:12.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015573799999999999 2024-04-24T14:35:12.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020306+0000", 2024-04-24T14:35:12.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044199399999999998 2024-04-24T14:35:12.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021887+0000", 2024-04-24T14:35:12.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015809400000000001 2024-04-24T14:35:12.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021942+0000", 2024-04-24T14:35:12.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5071000000000002e-05 2024-04-24T14:35:12.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.022007+0000", 2024-04-24T14:35:12.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5110000000000005e-05 2024-04-24T14:35:12.132 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.132 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.132 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.132 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.5324.0:127 3.0 e87/69 ondisk, result = 0)", 2024-04-24T14:35:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.020390921000001, 2024-04-24T14:35:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011245125, 2024-04-24T14:35:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:12.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:12.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007804+0000", 2024-04-24T14:35:12.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6610000000000003e-06 2024-04-24T14:35:12.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007817+0000", 2024-04-24T14:35:12.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3030999999999999e-05 2024-04-24T14:35:12.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007832+0000", 2024-04-24T14:35:12.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.47e-05 2024-04-24T14:35:12.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018843+0000", 2024-04-24T14:35:12.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011010647 2024-04-24T14:35:12.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018853+0000", 2024-04-24T14:35:12.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319000000000001e-05 2024-04-24T14:35:12.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019043+0000", 2024-04-24T14:35:12.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019055600000000001 2024-04-24T14:35:12.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.136 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.136 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.137 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:12.138 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd1 2024-04-24T14:35:12.139 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:35:12.139 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:35:12.139 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:35:12.139 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:35:12.139 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd1'\'']/id' 2024-04-24T14:35:12.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:35:12.140 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:12.140+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:12.163 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=20206 2024-04-24T14:35:12.163 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 20206 2024-04-24T14:35:12.163 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 20206 -C rbd-nbd 2024-04-24T14:35:12.170 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:35:12.171 INFO:tasks.workunit.client.0.smithi173.stdout: 20206 ? 00:00:00 rbd-nbd 2024-04-24T14:35:12.172 INFO:tasks.workunit.client.0.smithi173.stderr:++ ps -p 20206 -T 2024-04-24T14:35:12.172 INFO:tasks.workunit.client.0.smithi173.stderr:++ grep -c io_context_pool 2024-04-24T14:35:12.181 INFO:tasks.workunit.client.0.smithi173.stderr:+ ACTUAL=2 2024-04-24T14:35:12.181 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 2 -eq 2 ']' 2024-04-24T14:35:12.181 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device /dev/nbd1 20206 2024-04-24T14:35:12.181 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=/dev/nbd1 2024-04-24T14:35:12.181 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=20206 2024-04-24T14:35:12.181 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T14:35:12.182 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:35:12.182 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:35:12.182 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:35:12.183 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:35:12.185 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:35:12.185 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:35:12.185 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T14:35:12.238 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:12.242+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:12.247 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:12.247 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:12.247 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:35:12.247 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:12.256 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:35:12.287 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:35:12.287 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^20206\b' 2024-04-24T14:35:12.287 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^20206\b' 2024-04-24T14:35:12.339 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:12.342+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:12.352 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:35:12.353 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:35:12.353 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *20206\b' 2024-04-24T14:35:12.353 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *20206\b' 2024-04-24T14:35:12.365 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:35:12.366 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:35:12.401 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:12.401 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:12.402 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:12.402 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:12.402 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:12.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:35:12.440 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:12.444+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:12.511 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.758010338999995, 2024-04-24T14:35:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:35:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:35:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:35:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:35:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:35:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:35:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:35:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:35:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:35:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:35:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:35:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:35:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:35:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:35:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:35:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:35:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:35:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:35:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:35:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:35:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:35:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:35:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:35:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.757959505000002, 2024-04-24T14:35:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:35:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:35:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:35:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:35:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:35:12.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:35:12.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:35:12.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:35:12.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:35:12.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:35:12.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:12.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:35:12.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:35:12.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:35:12.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:35:12.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:12.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:35:12.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:35:12.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:35:12.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:35:12.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:35:12.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:35:12.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:35:12.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:35:12.525 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.525 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.525 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.525 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:12.526 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:12.526 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.757923198, 2024-04-24T14:35:12.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:35:12.526 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.526 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.526 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.526 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:12.526 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:12.526 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:35:12.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.526 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:12.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:12.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:35:12.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:35:12.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:35:12.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:35:12.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:35:12.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:35:12.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:35:12.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:35:12.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:35:12.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:12.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:35:12.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:35:12.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:35:12.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:35:12.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:12.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:35:12.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:35:12.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:35:12.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:35:12.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:35:12.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:35:12.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:35:12.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:35:12.532 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.532 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.532 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.533 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:12.533 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:12.533 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.754511948000001, 2024-04-24T14:35:12.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:35:12.533 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.533 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.533 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.533 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:12.533 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:12.534 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:35:12.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.534 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:12.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:12.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:35:12.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:12.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:35:12.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:35:12.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:35:12.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:35:12.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:35:12.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:35:12.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:35:12.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:35:12.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:12.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:35:12.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:35:12.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:35:12.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:35:12.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:12.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:35:12.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:35:12.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:35:12.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:35:12.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:35:12.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:35:12.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:35:12.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:35:12.540 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.540 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.540 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.540 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:12.540 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:12.540 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.754450270999996, 2024-04-24T14:35:12.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:35:12.541 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.541 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.541 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.541 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:12.541 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:12.541 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:35:12.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.541 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:12.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:12.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:35:12.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:35:12.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:35:12.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:35:12.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:35:12.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:35:12.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:35:12.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:35:12.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:35:12.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:12.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:35:12.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:35:12.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:35:12.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:35:12.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:12.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:35:12.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:35:12.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:35:12.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:35:12.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:35:12.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:35:12.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:35:12.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:35:12.547 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.547 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.547 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.547 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:12.548 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:12.548 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.754415695000006, 2024-04-24T14:35:12.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:35:12.548 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.548 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:12.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:12.548 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:35:12.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.548 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:12.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:12.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:12.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:35:12.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:35:12.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:35:12.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:35:12.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:35:12.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:35:12.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:35:12.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:35:12.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:12.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:35:12.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:35:12.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:35:12.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:35:12.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:12.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:35:12.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:35:12.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:35:12.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:35:12.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:35:12.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:35:12.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:35:12.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:35:12.554 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.554 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.554 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.555 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:12.555 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:12.555 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.754386355999998, 2024-04-24T14:35:12.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:35:12.555 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.555 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.555 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.555 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:12.555 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:12.556 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:35:12.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.556 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:12.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:12.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:35:12.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:35:12.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:35:12.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:35:12.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:35:12.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:35:12.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:35:12.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:35:12.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:35:12.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:12.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:35:12.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:35:12.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:35:12.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:35:12.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:12.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:35:12.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:35:12.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:35:12.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:35:12.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:35:12.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:35:12.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:35:12.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:35:12.561 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.562 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.562 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:12.562 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:12.562 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.754309774000006, 2024-04-24T14:35:12.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:35:12.562 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.562 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.563 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:12.563 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:12.563 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:35:12.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.563 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:12.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:12.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:35:12.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:35:12.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:35:12.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:35:12.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:35:12.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:35:12.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:35:12.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:35:12.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:35:12.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:12.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:35:12.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:35:12.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:35:12.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:35:12.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:12.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:35:12.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:35:12.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:35:12.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:35:12.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:35:12.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:35:12.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:35:12.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:35:12.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.569 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.569 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:12.569 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:12.569 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.754262229000005, 2024-04-24T14:35:12.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:35:12.570 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.570 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:12.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:12.570 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:35:12.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.570 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:12.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:12.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:35:12.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:35:12.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:35:12.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:35:12.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:35:12.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:35:12.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:35:12.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:35:12.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:35:12.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:12.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:35:12.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:35:12.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:35:12.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:35:12.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:12.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:35:12.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:35:12.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:35:12.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:35:12.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:35:12.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:35:12.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:35:12.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:35:12.576 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.576 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.576 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.576 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:12.577 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:12.577 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.754221587000004, 2024-04-24T14:35:12.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:35:12.577 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.577 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.577 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.577 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:12.577 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:12.577 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:35:12.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.577 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:12.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:12.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:12.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:35:12.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:35:12.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:35:12.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:35:12.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:35:12.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:35:12.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:35:12.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:35:12.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:12.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:35:12.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:35:12.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:35:12.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:35:12.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:12.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:35:12.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:35:12.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:35:12.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:35:12.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:35:12.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:35:12.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:35:12.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:35:12.583 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.583 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.583 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.584 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:12.584 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:12.584 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.754204475999998, 2024-04-24T14:35:12.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:35:12.584 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.584 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:12.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:12.584 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:35:12.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:35:12.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:35:12.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:35:12.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:35:12.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:35:12.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:35:12.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:35:12.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:35:12.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:35:12.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:12.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:35:12.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:35:12.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:35:12.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:35:12.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:35:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:35:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:35:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:35:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:35:12.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:35:12.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:35:12.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:35:12.590 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.590 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.590 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.590 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:103 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.554638363000002, 2024-04-24T14:35:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032646351999999997, 2024-04-24T14:35:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 103 2024-04-24T14:35:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:12.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:12.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959674+0000", 2024-04-24T14:35:12.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:12.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959688+0000", 2024-04-24T14:35:12.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2024-04-24T14:35:12.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959722+0000", 2024-04-24T14:35:12.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3936000000000002e-05 2024-04-24T14:35:12.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959735+0000", 2024-04-24T14:35:12.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2926e-05 2024-04-24T14:35:12.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959921+0000", 2024-04-24T14:35:12.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185601 2024-04-24T14:35:12.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960324+0000", 2024-04-24T14:35:12.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040356700000000002 2024-04-24T14:35:12.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:12.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960839+0000", 2024-04-24T14:35:12.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051410599999999996 2024-04-24T14:35:12.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.975634+0000", 2024-04-24T14:35:12.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014795770999999999 2024-04-24T14:35:12.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992113+0000", 2024-04-24T14:35:12.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016478982999999999 2024-04-24T14:35:12.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992228+0000", 2024-04-24T14:35:12.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114295 2024-04-24T14:35:12.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992323+0000", 2024-04-24T14:35:12.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5344000000000001e-05 2024-04-24T14:35:12.597 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.597 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.597 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.597 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:105 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.597 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:12.598 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.554388584999998, 2024-04-24T14:35:12.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032581757000000003, 2024-04-24T14:35:12.598 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.598 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.598 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.598 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.598 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.598 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105 2024-04-24T14:35:12.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.598 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:12.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:12.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959925+0000", 2024-04-24T14:35:12.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:12.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959937+0000", 2024-04-24T14:35:12.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2391999999999999e-05 2024-04-24T14:35:12.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:35:12.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2535999999999999e-05 2024-04-24T14:35:12.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959970+0000", 2024-04-24T14:35:12.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1039999999999999e-05 2024-04-24T14:35:12.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969531+0000", 2024-04-24T14:35:12.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095603949999999993 2024-04-24T14:35:12.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969669+0000", 2024-04-24T14:35:12.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013806499999999999 2024-04-24T14:35:12.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:12.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970010+0000", 2024-04-24T14:35:12.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034062200000000001 2024-04-24T14:35:12.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976983+0000", 2024-04-24T14:35:12.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069735600000000002 2024-04-24T14:35:12.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992413+0000", 2024-04-24T14:35:12.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015429841999999999 2024-04-24T14:35:12.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992457+0000", 2024-04-24T14:35:12.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4169999999999999e-05 2024-04-24T14:35:12.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992508+0000", 2024-04-24T14:35:12.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0995000000000002e-05 2024-04-24T14:35:12.604 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.604 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.604 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.605 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:113 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.605 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:12.605 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.534162293999998, 2024-04-24T14:35:12.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014793863000000001, 2024-04-24T14:35:12.605 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.605 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.605 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.605 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.605 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.606 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 113 2024-04-24T14:35:12.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.606 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:12.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:12.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980152+0000", 2024-04-24T14:35:12.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980156+0000", 2024-04-24T14:35:12.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.619e-06 2024-04-24T14:35:12.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980163+0000", 2024-04-24T14:35:12.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7070000000000003e-06 2024-04-24T14:35:12.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980173+0000", 2024-04-24T14:35:12.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0957e-05 2024-04-24T14:35:12.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983381+0000", 2024-04-24T14:35:12.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032079930000000001 2024-04-24T14:35:12.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983628+0000", 2024-04-24T14:35:12.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024665700000000001 2024-04-24T14:35:12.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:12.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983964+0000", 2024-04-24T14:35:12.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000336332 2024-04-24T14:35:12.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992694+0000", 2024-04-24T14:35:12.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087297009999999994 2024-04-24T14:35:12.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994878+0000", 2024-04-24T14:35:12.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021841439999999998 2024-04-24T14:35:12.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994917+0000", 2024-04-24T14:35:12.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8544999999999998e-05 2024-04-24T14:35:12.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994947+0000", 2024-04-24T14:35:12.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9705000000000001e-05 2024-04-24T14:35:12.612 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.612 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.612 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.612 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:130 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.612 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:12.612 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.508468858999997, 2024-04-24T14:35:12.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014646289999999999, 2024-04-24T14:35:12.612 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.612 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.612 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.613 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.613 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.613 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 130 2024-04-24T14:35:12.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.613 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:12.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:12.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005844+0000", 2024-04-24T14:35:12.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:12.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005853+0000", 2024-04-24T14:35:12.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1319999999999994e-06 2024-04-24T14:35:12.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005870+0000", 2024-04-24T14:35:12.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7448999999999998e-05 2024-04-24T14:35:12.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005876+0000", 2024-04-24T14:35:12.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3319999999999997e-06 2024-04-24T14:35:12.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005913+0000", 2024-04-24T14:35:12.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6835999999999998e-05 2024-04-24T14:35:12.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006017+0000", 2024-04-24T14:35:12.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010358500000000001 2024-04-24T14:35:12.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:12.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006139+0000", 2024-04-24T14:35:12.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001227 2024-04-24T14:35:12.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.010194+0000", 2024-04-24T14:35:12.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040549380000000001 2024-04-24T14:35:12.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020365+0000", 2024-04-24T14:35:12.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010170983 2024-04-24T14:35:12.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020428+0000", 2024-04-24T14:35:12.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2457000000000002e-05 2024-04-24T14:35:12.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020492+0000", 2024-04-24T14:35:12.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4592999999999996e-05 2024-04-24T14:35:12.619 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.619 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.619 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.619 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:155 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1601536~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.619 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:12.619 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.474808072000002, 2024-04-24T14:35:12.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015144391, 2024-04-24T14:35:12.619 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.619 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.620 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.620 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.620 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.620 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 155 2024-04-24T14:35:12.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.620 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:12.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:12.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039506+0000", 2024-04-24T14:35:12.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:12.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039766+0000", 2024-04-24T14:35:12.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025994400000000002 2024-04-24T14:35:12.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039780+0000", 2024-04-24T14:35:12.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3366e-05 2024-04-24T14:35:12.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039787+0000", 2024-04-24T14:35:12.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4359999999999996e-06 2024-04-24T14:35:12.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.050886+0000", 2024-04-24T14:35:12.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011098993 2024-04-24T14:35:12.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051026+0000", 2024-04-24T14:35:12.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001395 2024-04-24T14:35:12.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:12.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051211+0000", 2024-04-24T14:35:12.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018547099999999999 2024-04-24T14:35:12.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054490+0000", 2024-04-24T14:35:12.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032785499999999999 2024-04-24T14:35:12.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054505+0000", 2024-04-24T14:35:12.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5906999999999999e-05 2024-04-24T14:35:12.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054594+0000", 2024-04-24T14:35:12.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8817e-05 2024-04-24T14:35:12.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054651+0000", 2024-04-24T14:35:12.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7040999999999998e-05 2024-04-24T14:35:12.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.626 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.626 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:156 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1732608~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.626 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:12.626 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.474517366000001, 2024-04-24T14:35:12.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016895194999999998, 2024-04-24T14:35:12.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.627 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.627 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.627 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.627 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.627 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 156 2024-04-24T14:35:12.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.627 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:12.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:12.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039797+0000", 2024-04-24T14:35:12.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040065+0000", 2024-04-24T14:35:12.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000268361 2024-04-24T14:35:12.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040086+0000", 2024-04-24T14:35:12.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1024000000000001e-05 2024-04-24T14:35:12.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040094+0000", 2024-04-24T14:35:12.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9379999999999992e-06 2024-04-24T14:35:12.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051936+0000", 2024-04-24T14:35:12.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011842518 2024-04-24T14:35:12.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052062+0000", 2024-04-24T14:35:12.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125482 2024-04-24T14:35:12.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:12.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052206+0000", 2024-04-24T14:35:12.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143953 2024-04-24T14:35:12.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054715+0000", 2024-04-24T14:35:12.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025088240000000002 2024-04-24T14:35:12.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056534+0000", 2024-04-24T14:35:12.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001819436 2024-04-24T14:35:12.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056610+0000", 2024-04-24T14:35:12.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5383000000000007e-05 2024-04-24T14:35:12.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056693+0000", 2024-04-24T14:35:12.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3283999999999997e-05 2024-04-24T14:35:12.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.633 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.633 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:157 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1863680~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.634 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:12.634 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.474209987000002, 2024-04-24T14:35:12.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016763674999999999, 2024-04-24T14:35:12.634 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.634 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.634 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157 2024-04-24T14:35:12.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.634 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:12.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:12.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040104+0000", 2024-04-24T14:35:12.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040244+0000", 2024-04-24T14:35:12.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014035499999999999 2024-04-24T14:35:12.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040255+0000", 2024-04-24T14:35:12.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0135999999999999e-05 2024-04-24T14:35:12.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040260+0000", 2024-04-24T14:35:12.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7919999999999998e-06 2024-04-24T14:35:12.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053406+0000", 2024-04-24T14:35:12.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013146023999999999 2024-04-24T14:35:12.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053531+0000", 2024-04-24T14:35:12.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012471800000000001 2024-04-24T14:35:12.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:12.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053715+0000", 2024-04-24T14:35:12.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018395099999999999 2024-04-24T14:35:12.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056711+0000", 2024-04-24T14:35:12.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029956520000000001 2024-04-24T14:35:12.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056783+0000", 2024-04-24T14:35:12.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2010999999999999e-05 2024-04-24T14:35:12.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056831+0000", 2024-04-24T14:35:12.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8189999999999998e-05 2024-04-24T14:35:12.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056869+0000", 2024-04-24T14:35:12.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7812000000000003e-05 2024-04-24T14:35:12.640 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.640 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.640 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.640 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:158 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1994752~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.641 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:12.641 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.474048060000001, 2024-04-24T14:35:12.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016845334999999999, 2024-04-24T14:35:12.641 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.641 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:12.641 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.641 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.641 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.641 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2024-04-24T14:35:12.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.641 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:12.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:12.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040266+0000", 2024-04-24T14:35:12.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040379+0000", 2024-04-24T14:35:12.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112728 2024-04-24T14:35:12.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040387+0000", 2024-04-24T14:35:12.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8499999999999994e-06 2024-04-24T14:35:12.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040391+0000", 2024-04-24T14:35:12.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2719999999999999e-06 2024-04-24T14:35:12.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054803+0000", 2024-04-24T14:35:12.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014411548 2024-04-24T14:35:12.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054982+0000", 2024-04-24T14:35:12.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017941699999999999 2024-04-24T14:35:12.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:12.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.055167+0000", 2024-04-24T14:35:12.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018542100000000001 2024-04-24T14:35:12.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:12.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056718+0000", 2024-04-24T14:35:12.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015505670000000001 2024-04-24T14:35:12.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:12.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057020+0000", 2024-04-24T14:35:12.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030227500000000002 2024-04-24T14:35:12.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057070+0000", 2024-04-24T14:35:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9611000000000003e-05 2024-04-24T14:35:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057112+0000", 2024-04-24T14:35:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2373999999999998e-05 2024-04-24T14:35:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:159 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.468613071999997, 2024-04-24T14:35:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015930650000000001, 2024-04-24T14:35:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2024-04-24T14:35:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045701+0000", 2024-04-24T14:35:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045707+0000", 2024-04-24T14:35:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.562e-06 2024-04-24T14:35:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045728+0000", 2024-04-24T14:35:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1109999999999999e-05 2024-04-24T14:35:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045738+0000", 2024-04-24T14:35:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0016e-05 2024-04-24T14:35:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056183+0000", 2024-04-24T14:35:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010444836000000001 2024-04-24T14:35:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056470+0000", 2024-04-24T14:35:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028696899999999998 2024-04-24T14:35:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057160+0000", 2024-04-24T14:35:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069004699999999995 2024-04-24T14:35:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057291+0000", 2024-04-24T14:35:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013096200000000001 2024-04-24T14:35:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.061633+0000", 2024-04-24T14:35:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004341414 2024-04-24T14:35:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.654 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:12.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:35:12.657 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:12.660+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:12.756 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:12.758+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:12.765 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:35:12.765 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:12.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:35:12.857 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:12.858+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:12.867 INFO:tasks.workunit.client.0.smithi173.stderr:+ EXPECTED=7 2024-04-24T14:35:12.868 INFO:tasks.workunit.client.0.smithi173.stderr:++ _sudo rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd16809 2024-04-24T14:35:12.868 INFO:tasks.workunit.client.0.smithi173.stderr:++ local cmd 2024-04-24T14:35:12.868 INFO:tasks.workunit.client.0.smithi173.stderr:+++ id -u 2024-04-24T14:35:12.870 INFO:tasks.workunit.client.0.smithi173.stderr:++ '[' 1000 -eq 0 ']' 2024-04-24T14:35:12.871 INFO:tasks.workunit.client.0.smithi173.stderr:+++ which rbd 2024-04-24T14:35:12.874 INFO:tasks.workunit.client.0.smithi173.stderr:++ cmd=/usr/bin/rbd 2024-04-24T14:35:12.874 INFO:tasks.workunit.client.0.smithi173.stderr:++ shift 2024-04-24T14:35:12.874 INFO:tasks.workunit.client.0.smithi173.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd16809 2024-04-24T14:35:12.876 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:12.876 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:12.876 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:12.876 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:12.876 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:12.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:35:12.958 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:12.962+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:12.991 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:12.991 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:12.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:12.991 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.523962181000002, 2024-04-24T14:35:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:35:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:35:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:12.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:12.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:35:12.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:35:12.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:35:12.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:35:12.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:35:12.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:35:12.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:35:12.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:35:12.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:35:12.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:12.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:35:12.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:35:12.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:12.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:35:12.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:35:12.996 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.996 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:12.996 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:12.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.996 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:12.996 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:12.996 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.148605477999993, 2024-04-24T14:35:12.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:35:12.996 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:12.996 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:12.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:12.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:12.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:35:12.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:12.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:12.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:35:12.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:35:12.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:35:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:35:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:35:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:35:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:35:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:35:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:12.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:35:13.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:35:13.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:35:13.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:35:13.000 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.000 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.000 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.000 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:13.001 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:13.001 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.147177323999998, 2024-04-24T14:35:13.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:35:13.001 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.001 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.001 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.001 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:13.001 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:13.001 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:35:13.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.001 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:13.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:13.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:35:13.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:13.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:35:13.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:35:13.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:35:13.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:35:13.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:35:13.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:35:13.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:35:13.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:35:13.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:35:13.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:35:13.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:35:13.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:35:13.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.005 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.005 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:13.005 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:13.005 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.147150234999998, 2024-04-24T14:35:13.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:35:13.005 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.005 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:13.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:13.006 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:35:13.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.006 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:13.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:13.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:35:13.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:13.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:35:13.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:35:13.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:35:13.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:35:13.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:35:13.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:35:13.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:35:13.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:35:13.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:35:13.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:35:13.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:35:13.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:35:13.009 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.009 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.009 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.009 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:13.009 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:13.010 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.147126267999994, 2024-04-24T14:35:13.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:35:13.010 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.010 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.010 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.010 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:13.010 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:13.010 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:35:13.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.010 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:35:13.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:35:13.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:35:13.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:35:13.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:35:13.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:35:13.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:35:13.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:35:13.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:35:13.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:35:13.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:35:13.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:35:13.014 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.014 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.014 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.014 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:13.014 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:13.014 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.133142289000006, 2024-04-24T14:35:13.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:35:13.014 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.014 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:35:13.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:13.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:35:13.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:35:13.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:35:13.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:35:13.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:35:13.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:35:13.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:35:13.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:35:13.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:35:13.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:35:13.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:35:13.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:35:13.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:35:13.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:35:13.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:35:13.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:35:13.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:35:13.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:35:13.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:35:13.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:35:13.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.020 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.020 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.020 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:13.020 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:13.020 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.133025978000006, 2024-04-24T14:35:13.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:35:13.020 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.020 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:13.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:13.020 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:35:13.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.021 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:13.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:13.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:35:13.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:13.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:35:13.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:35:13.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:35:13.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:35:13.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:35:13.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:35:13.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:35:13.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:35:13.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:35:13.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:35:13.025 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.025 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.025 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.026 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:13.026 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:13.026 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.132978320000007, 2024-04-24T14:35:13.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:35:13.026 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.026 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:13.026 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:13.026 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:35:13.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.026 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:13.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:13.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:35:13.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:35:13.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:35:13.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:35:13.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:35:13.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:35:13.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:35:13.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:35:13.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:35:13.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:35:13.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:35:13.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:35:13.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:35:13.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:35:13.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:35:13.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:35:13.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:35:13.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.031 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.032 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:13.032 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:13.032 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.128941022000006, 2024-04-24T14:35:13.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:35:13.032 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.032 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:13.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:13.032 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:35:13.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.032 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:13.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:13.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:35:13.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:13.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:35:13.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:35:13.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:35:13.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:35:13.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:35:13.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:35:13.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:35:13.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:35:13.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:35:13.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:35:13.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:35:13.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:35:13.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:35:13.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:35:13.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:35:13.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:35:13.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:35:13.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:35:13.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:35:13.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:35:13.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.037 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.038 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:13.038 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:13.038 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.128877263000007, 2024-04-24T14:35:13.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:35:13.038 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.038 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.038 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.038 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:13.038 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:13.038 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:35:13.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.039 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:13.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:13.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:35:13.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:13.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:35:13.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:35:13.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:35:13.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:35:13.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:35:13.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:35:13.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:35:13.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:35:13.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:35:13.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:35:13.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:35:13.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:35:13.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:35:13.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:35:13.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:35:13.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:35:13.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:35:13.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:35:13.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:35:13.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:35:13.043 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.043 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.044 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:13.044 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:13.044 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.128851772999994, 2024-04-24T14:35:13.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:35:13.044 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.044 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.044 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.044 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:13.044 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:13.044 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:35:13.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.045 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:13.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:13.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:13.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:13.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:35:13.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:35:13.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:35:13.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:35:13.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:35:13.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:35:13.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:35:13.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:35:13.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:35:13.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:35:13.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:35:13.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:35:13.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:35:13.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:35:13.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:35:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:35:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:35:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:35:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:35:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:35:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.049 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.050 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:13.050 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:13.050 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.128828022999997, 2024-04-24T14:35:13.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:35:13.050 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.050 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:13.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:13.050 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:35:13.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.051 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:13.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:13.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:35:13.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:13.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:35:13.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:35:13.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:35:13.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:35:13.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:35:13.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:35:13.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:35:13.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:35:13.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:35:13.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:35:13.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:35:13.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:35:13.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:35:13.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:35:13.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:35:13.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:35:13.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:35:13.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:35:13.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:35:13.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:35:13.055 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.055 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.055 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.056 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:13.056 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:13.056 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.128789510000004, 2024-04-24T14:35:13.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:35:13.056 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.056 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:13.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:13.056 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:35:13.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.056 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:13.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:13.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:35:13.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:13.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:35:13.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:35:13.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:35:13.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:35:13.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:35:13.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:35:13.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:35:13.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:35:13.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:35:13.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:35:13.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:35:13.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:35:13.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:35:13.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:35:13.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:35:13.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:35:13.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:35:13.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:35:13.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:35:13.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:35:13.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.061 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.061 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.128761312999998, 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:13.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:13.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:35:13.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:13.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:35:13.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:35:13.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:35:13.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:35:13.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:35:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:35:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:35:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:35:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:35:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:35:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:35:13.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:35:13.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:35:13.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:35:13.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:35:13.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:35:13.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:35:13.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:35:13.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:35:13.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:35:13.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.067 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.068 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:13.068 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:13.068 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.128739855999996, 2024-04-24T14:35:13.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:35:13.068 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.068 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:13.068 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:13.068 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:35:13.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:35:13.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:13.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:35:13.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:35:13.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:35:13.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:35:13.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:35:13.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:35:13.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:35:13.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:35:13.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:35:13.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:35:13.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:35:13.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:35:13.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:35:13.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:35:13.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.073 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.073 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:13.073 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:13.074 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.128721014000007, 2024-04-24T14:35:13.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:35:13.074 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.074 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:13.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:13.074 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:35:13.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.074 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:35:13.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:35:13.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:35:13.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:35:13.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:35:13.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:35:13.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:35:13.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:35:13.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:35:13.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:35:13.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:35:13.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:35:13.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:35:13.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:35:13.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:35:13.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:35:13.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.915888864999999, 2024-04-24T14:35:13.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:35:13.080 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.080 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:13.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:35:13.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:13.080 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:35:13.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.080 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:13.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:13.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:35:13.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:13.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:35:13.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:35:13.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:35:13.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:35:13.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:35:13.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:35:13.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:35:13.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:35:13.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:35:13.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:35:13.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:35:13.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:35:13.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:35:13.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:35:13.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:35:13.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:35:13.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:35:13.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:35:13.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:35:13.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:35:13.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.085 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.085 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:39 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1753088~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:13.085 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:13.085 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.976116449000003, 2024-04-24T14:35:13.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01139394, 2024-04-24T14:35:13.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.086 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:13.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:13.086 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 39 2024-04-24T14:35:13.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.086 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:13.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:13.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018967+0000", 2024-04-24T14:35:13.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:13.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019242+0000", 2024-04-24T14:35:13.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027504500000000002 2024-04-24T14:35:13.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019274+0000", 2024-04-24T14:35:13.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2472999999999999e-05 2024-04-24T14:35:13.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019288+0000", 2024-04-24T14:35:13.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3283e-05 2024-04-24T14:35:13.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026753+0000", 2024-04-24T14:35:13.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074651120000000003 2024-04-24T14:35:13.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026971+0000", 2024-04-24T14:35:13.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021810800000000001 2024-04-24T14:35:13.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:13.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.027211+0000", 2024-04-24T14:35:13.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023975 2024-04-24T14:35:13.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030200+0000", 2024-04-24T14:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029890709999999998 2024-04-24T14:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030252+0000", 2024-04-24T14:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1926999999999999e-05 2024-04-24T14:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030312+0000", 2024-04-24T14:35:13.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0602e-05 2024-04-24T14:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030363+0000", 2024-04-24T14:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0342e-05 2024-04-24T14:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:40 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1884160~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:13.091 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.975780475000001, 2024-04-24T14:35:13.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011199064, 2024-04-24T14:35:13.092 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.092 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:13.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:13.092 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 40 2024-04-24T14:35:13.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.092 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:13.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:13.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019303+0000", 2024-04-24T14:35:13.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:13.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019729+0000", 2024-04-24T14:35:13.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042620899999999998 2024-04-24T14:35:13.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019769+0000", 2024-04-24T14:35:13.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9847e-05 2024-04-24T14:35:13.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019797+0000", 2024-04-24T14:35:13.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8347000000000002e-05 2024-04-24T14:35:13.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028299+0000", 2024-04-24T14:35:13.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085010629999999997 2024-04-24T14:35:13.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028385+0000", 2024-04-24T14:35:13.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6824999999999996e-05 2024-04-24T14:35:13.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:13.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028510+0000", 2024-04-24T14:35:13.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012483000000000001 2024-04-24T14:35:13.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:13.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030410+0000", 2024-04-24T14:35:13.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001899612 2024-04-24T14:35:13.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:13.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030416+0000", 2024-04-24T14:35:13.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7699999999999998e-06 2024-04-24T14:35:13.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:13.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030461+0000", 2024-04-24T14:35:13.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5157999999999997e-05 2024-04-24T14:35:13.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030504+0000", 2024-04-24T14:35:13.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2988999999999998e-05 2024-04-24T14:35:13.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.097 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.098 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:41 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 2015232~81920] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:13.098 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:13.098 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.975271767999999, 2024-04-24T14:35:13.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012345257, 2024-04-24T14:35:13.098 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:13.098 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:13.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:13.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:13.098 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 41 2024-04-24T14:35:13.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.098 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:13.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:13.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:13.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:13.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:13.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:13.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:13.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019812+0000", 2024-04-24T14:35:13.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:13.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:13.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019975+0000", 2024-04-24T14:35:13.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016325700000000001 2024-04-24T14:35:13.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:13.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020025+0000", 2024-04-24T14:35:13.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9484999999999998e-05 2024-04-24T14:35:13.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020035+0000", 2024-04-24T14:35:13.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0444e-05 2024-04-24T14:35:13.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:13.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029150+0000", 2024-04-24T14:35:13.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091150160000000001 2024-04-24T14:35:13.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:13.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029352+0000", 2024-04-24T14:35:13.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020166499999999999 2024-04-24T14:35:13.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:13.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029611+0000", 2024-04-24T14:35:13.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025949600000000001 2024-04-24T14:35:13.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:13.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030518+0000", 2024-04-24T14:35:13.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090633300000000005 2024-04-24T14:35:13.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:13.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032088+0000", 2024-04-24T14:35:13.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015703290000000001 2024-04-24T14:35:13.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:13.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032130+0000", 2024-04-24T14:35:13.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2273e-05 2024-04-24T14:35:13.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:13.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:13.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:13.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032159+0000", 2024-04-24T14:35:13.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8387000000000001e-05 2024-04-24T14:35:13.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:13.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:13.104 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:13.105 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:35:13.106 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:13.108+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:13.107 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV=/dev/nbd1 2024-04-24T14:35:13.107 INFO:tasks.workunit.client.0.smithi173.stderr:+ get_pid rbd 2024-04-24T14:35:13.107 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pool=rbd 2024-04-24T14:35:13.107 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns= 2024-04-24T14:35:13.107 INFO:tasks.workunit.client.0.smithi173.stderr:++ rbd device --device-type nbd --format xml list 2024-04-24T14:35:13.107 INFO:tasks.workunit.client.0.smithi173.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd16809'\''][device='\''/dev/nbd1'\'']/id' 2024-04-24T14:35:13.190 INFO:tasks.workunit.client.0.smithi173.stderr:+ PID=20273 2024-04-24T14:35:13.190 INFO:tasks.workunit.client.0.smithi173.stderr:+ test -n 20273 2024-04-24T14:35:13.190 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -p 20273 -C rbd-nbd 2024-04-24T14:35:13.200 INFO:tasks.workunit.client.0.smithi173.stdout: PID TTY TIME CMD 2024-04-24T14:35:13.200 INFO:tasks.workunit.client.0.smithi173.stdout: 20273 ? 00:00:00 rbd-nbd 2024-04-24T14:35:13.200 INFO:tasks.workunit.client.0.smithi173.stderr:++ grep -c io_context_pool 2024-04-24T14:35:13.201 INFO:tasks.workunit.client.0.smithi173.stderr:++ ps -p 20273 -T 2024-04-24T14:35:13.206 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:13.208+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:13.212 INFO:tasks.workunit.client.0.smithi173.stderr:+ ACTUAL=7 2024-04-24T14:35:13.212 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 7 -eq 7 ']' 2024-04-24T14:35:13.212 INFO:tasks.workunit.client.0.smithi173.stderr:+ unmap_device /dev/nbd1 20273 2024-04-24T14:35:13.212 INFO:tasks.workunit.client.0.smithi173.stderr:+ local args=/dev/nbd1 2024-04-24T14:35:13.213 INFO:tasks.workunit.client.0.smithi173.stderr:+ local pid=20273 2024-04-24T14:35:13.213 INFO:tasks.workunit.client.0.smithi173.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T14:35:13.213 INFO:tasks.workunit.client.0.smithi173.stderr:+ local cmd 2024-04-24T14:35:13.213 INFO:tasks.workunit.client.0.smithi173.stderr:++ id -u 2024-04-24T14:35:13.213 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' 1000 -eq 0 ']' 2024-04-24T14:35:13.214 INFO:tasks.workunit.client.0.smithi173.stderr:++ which rbd 2024-04-24T14:35:13.216 INFO:tasks.workunit.client.0.smithi173.stderr:+ cmd=/usr/bin/rbd 2024-04-24T14:35:13.216 INFO:tasks.workunit.client.0.smithi173.stderr:+ shift 2024-04-24T14:35:13.216 INFO:tasks.workunit.client.0.smithi173.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-04-24T14:35:13.219 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:13.219 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:13.220 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:35:13.220 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:13.229 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:35:13.307 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:13.310+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:13.318 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd device --device-type nbd list 2024-04-24T14:35:13.319 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^20273\b' 2024-04-24T14:35:13.319 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^20273\b' 2024-04-24T14:35:13.326 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:13.326 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:13.327 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:35:13.327 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:35:13.327 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:13.336 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:35:13.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T14:35:13.382 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:35:13.383 INFO:tasks.workunit.client.0.smithi173.stderr:+ ps -C rbd-nbd 2024-04-24T14:35:13.383 INFO:tasks.workunit.client.0.smithi173.stderr:+ expect_false grep '^ *20273\b' 2024-04-24T14:35:13.384 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep '^ *20273\b' 2024-04-24T14:35:13.393 INFO:tasks.workunit.client.0.smithi173.stderr:+ return 0 2024-04-24T14:35:13.393 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:35:13.408 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:13.412+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:13.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:35:13.436 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:35:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:35:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:35:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 170.812315281, 2024-04-24T14:35:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:35:13.436 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:35:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.437 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:35:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:35:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:35:13.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:35:13.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:35:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:35:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:35:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:35:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:35:13.440 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:35:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:35:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:35:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:35:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:35:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.441 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:35:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:35:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 170.812138194, 2024-04-24T14:35:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:35:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.442 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:35:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:13.443 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:35:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:35:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:35:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.444 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:35:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:35:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:35:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:35:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:35:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:35:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:35:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:35:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:35:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:35:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:35:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:35:13.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:35:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:35:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:35:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:35:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:35:13.447 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:35:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 170.81203428000001, 2024-04-24T14:35:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:35:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.448 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.449 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:35:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:35:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:35:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:35:13.450 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:35:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:35:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:35:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:35:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:35:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.451 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:35:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:35:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:35:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:35:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:35:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:35:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.452 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.453 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 170.81164582900001, 2024-04-24T14:35:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:35:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:13.454 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:35:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.455 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:35:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:35:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:35:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:35:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:35:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.456 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:35:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:35:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:35:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:35:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:35:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:35:13.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:35:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:35:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:35:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:35:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.458 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:35:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:35:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:35:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:35:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:35:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:35:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.459 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 170.811560617, 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.460 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:35:13.461 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:35:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:35:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:35:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:35:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.462 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:35:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:35:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:35:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:35:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:35:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:35:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.463 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:35:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:35:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:35:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:35:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:35:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:35:13.464 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:35:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:35:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:35:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:35:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.465 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 170.81132373599999, 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.466 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:35:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.467 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:35:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:35:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:35:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:35:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:35:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:35:13.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:35:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:35:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:35:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:35:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:35:13.469 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:35:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:35:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:35:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:35:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:35:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:13.470 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:35:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:35:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:35:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:35:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:13.471 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 170.811242214, 2024-04-24T14:35:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:35:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:35:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.472 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.474 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:35:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:35:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:35:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:35:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:35:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:35:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.475 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:35:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:35:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:35:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:35:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:35:13.476 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:35:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:35:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:35:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 170.379329867, 2024-04-24T14:35:13.477 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:35:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:35:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.478 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:35:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:35:13.479 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:35:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:35:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:35:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:35:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:35:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.480 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:35:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:35:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:35:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:35:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:35:13.481 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:35:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:35:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:35:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:35:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:35:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.482 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 170.379301567, 2024-04-24T14:35:13.483 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:35:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:35:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.484 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:35:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:35:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:35:13.485 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:35:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:35:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:35:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:35:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:35:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:35:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:35:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:35:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:35:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:35:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.487 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:35:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:35:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:35:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:35:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:35:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:35:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 170.37921386299999, 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.489 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:35:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:13.490 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:35:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:35:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:35:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.491 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.492 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:35:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:35:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:35:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:35:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:35:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:35:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.493 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:13.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:35:13.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:35:13.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:13.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:35:13.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:35:13.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.494 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.494 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.494 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:35:13.494 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:35:13.494 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.495 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.495 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.495 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.495 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.495 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:13.495 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:13.495 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 170.37909106199999, 2024-04-24T14:35:13.495 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:35:13.495 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.495 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.495 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.495 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:13.495 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:13.495 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:35:13.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.496 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:13.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.496 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.496 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:13.496 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.496 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:35:13.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:13.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:35:13.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:35:13.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.497 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.497 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.497 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:35:13.497 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:35:13.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:35:13.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:35:13.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:35:13.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:35:13.498 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.498 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.498 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.498 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:35:13.498 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:35:13.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:13.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:35:13.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:35:13.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:13.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:35:13.499 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:35:13.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.499 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.499 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:13.499 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.500 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.501 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.501 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:35:13.501 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:13.501 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 101.072846502, 2024-04-24T14:35:13.501 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:35:13.501 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.501 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:13.501 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.501 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:35:13.501 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:35:13.501 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:35:13.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.501 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:13.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:13.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.502 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.502 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.502 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:35:13.502 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:13.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:35:13.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:35:13.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.503 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:35:13.503 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:35:13.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.503 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.503 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.504 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:35:13.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:35:13.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:13.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:35:13.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:35:13.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.505 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.505 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.505 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:35:13.505 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:35:13.505 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.505 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.506 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.506 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.506 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:13.506 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:13.506 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 93.378722503000006, 2024-04-24T14:35:13.506 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:35:13.506 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.506 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:13.506 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.506 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:35:13.506 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:35:13.507 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:35:13.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.507 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:13.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.507 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.507 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:13.507 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.507 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:35:13.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:13.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:35:13.508 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:35:13.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.508 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.508 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.508 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:35:13.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:35:13.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:35:13.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:35:13.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.509 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:35:13.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:35:13.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.509 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:35:13.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:35:13.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:35:13.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:35:13.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:35:13.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:35:13.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:35:13.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:35:13.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:35:13.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:35:13.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:35:13.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:35:13.512 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.512 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.512 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.512 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:13.512 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:13.512 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.419362536999998, 2024-04-24T14:35:13.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:35:13.512 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.512 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:13.512 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.512 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.513 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:35:13.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:13.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:35:13.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:35:13.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.514 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.514 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:35:13.514 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:35:13.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:35:13.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:35:13.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.515 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:35:13.515 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:35:13.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.515 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.515 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:35:13.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:35:13.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:35:13.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:35:13.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.516 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:35:13.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:35:13.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.517 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.518 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.518 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:13.518 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:13.518 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.419281982000001, 2024-04-24T14:35:13.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:35:13.518 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.518 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:13.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:13.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:13.518 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:35:13.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.519 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:13.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:13.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:35:13.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:35:13.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:35:13.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:35:13.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:35:13.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.524 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:13.524 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:13.524 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.419234252999999, 2024-04-24T14:35:13.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:35:13.524 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.524 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:13.524 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.524 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:13.524 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:13.524 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:35:13.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.524 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:35:13.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:35:13.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:35:13.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:35:13.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:35:13.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:35:13.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:35:13.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:35:13.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:35:13.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:35:13.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.527 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.527 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:35:13.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:35:13.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:35:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:35:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:35:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:35:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:35:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:35:13.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:35:13.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:35:13.529 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.529 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.529 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.529 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:13.529 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:13.529 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.419206489999993, 2024-04-24T14:35:13.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:35:13.530 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.530 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:13.530 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.530 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:13.530 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:13.530 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:35:13.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.530 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:13.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:13.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:35:13.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:13.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:35:13.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:35:13.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:35:13.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:35:13.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:35:13.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:35:13.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:35:13.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:35:13.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:35:13.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:35:13.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:35:13.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:35:13.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:35:13.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:35:13.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:35:13.535 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.535 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.535 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.535 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:13.535 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:13.535 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.419167055000003, 2024-04-24T14:35:13.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:35:13.535 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.535 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:13.535 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:35:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:13.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:35:13.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:13.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:35:13.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:35:13.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:35:13.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:35:13.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:35:13.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:35:13.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:35:13.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:35:13.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:35:13.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:35:13.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:35:13.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:35:13.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:35:13.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:35:13.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:35:13.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:35:13.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.540 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.540 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.540 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:35:13.540 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:35:13.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.419142070999996, 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.541 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:13.542 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.542 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:13.542 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:13.542 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:35:13.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.542 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.542 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:13.542 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.542 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.542 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:13.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:35:13.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:13.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.543 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.543 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.543 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:35:13.543 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:35:13.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:35:13.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:35:13.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:35:13.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:35:13.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:35:13.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.545 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.545 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.545 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:35:13.545 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:35:13.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:35:13.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:35:13.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:35:13.546 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:35:13.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.546 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.546 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.546 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:35:13.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:35:13.547 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.547 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.547 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.547 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.547 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:13.547 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:13.547 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 74.419116411999994, 2024-04-24T14:35:13.547 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:35:13.547 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:13.547 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:13.547 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:13.548 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:13.548 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:13.548 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:35:13.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.548 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:13.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:13.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:13.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.548 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:13.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:35:13.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:35:13.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:35:13.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:35:13.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:13.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:35:13.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:35:13.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:13.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:35:13.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:35:13.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:13.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:35:13.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:35:13.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:13.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:13.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:13.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:35:13.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:35:13.552 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.552 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.552 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.553 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:13.553 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:13.553 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:13.554 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:35:13.555 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:13.556+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:13.654 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:13.656+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:13.683 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:35:13.702 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:35:13.704 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:13.705 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:35:13.705 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:13.714 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:35:13.721 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:35:13.745 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:35:13.763 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:13.768+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:13.774 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:35:13.794 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:35:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:35:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:35:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:35:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:35:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:35:13.797 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:35:13.818 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:13.818 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:13.827 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:35:13.857 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:13.858+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:13.895 INFO:tasks.workunit.client.0.smithi173.stderr:+ DEV= 2024-04-24T14:35:13.895 INFO:tasks.workunit.client.0.smithi173.stderr:+ echo OK 2024-04-24T14:35:13.896 INFO:tasks.workunit.client.0.smithi173.stdout:OK 2024-04-24T14:35:13.896 INFO:tasks.workunit.client.0.smithi173.stderr:+ cleanup 2024-04-24T14:35:13.896 INFO:tasks.workunit.client.0.smithi173.stderr:+ local ns s 2024-04-24T14:35:13.896 INFO:tasks.workunit.client.0.smithi173.stderr:+ set +e 2024-04-24T14:35:13.897 INFO:tasks.workunit.client.0.smithi173.stderr:+ mount 2024-04-24T14:35:13.897 INFO:tasks.workunit.client.0.smithi173.stderr:+ fgrep /tmp/tmp.waDYLkcsIV/mnt 2024-04-24T14:35:13.908 INFO:tasks.workunit.client.0.smithi173.stderr:+ rm -Rf /tmp/tmp.waDYLkcsIV 2024-04-24T14:35:13.922 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:13.922 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:13.924 INFO:tasks.workunit.client.0.smithi173.stderr:+ '[' -n '' ']' 2024-04-24T14:35:13.924 INFO:tasks.workunit.client.0.smithi173.stderr:+ for ns in '' ${NS} 2024-04-24T14:35:13.924 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd -p rbd --namespace '' status testrbdnbd16809 2024-04-24T14:35:13.931 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:35:13.957 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:13.960+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:13.987 INFO:tasks.workunit.client.0.smithi173.stdout:Watchers: none 2024-04-24T14:35:13.992 INFO:tasks.workunit.client.0.smithi173.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-04-24T14:35:13.993 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:35:14.033 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:14.034 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:14.043 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:35:14.058 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:14.062+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:14.159 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:14.162+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:14.185 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:14.185 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:14.194 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:35:14.260 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:14.264+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:14.297 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:14.297 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:14.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:35:14.360 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-24T14:35:14.360 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:14.364+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:14.418 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:14.418 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:14.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:35:14.461 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:14.462+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:14.495 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd -p rbd --namespace '' status testrbdnbd16809 2024-04-24T14:35:14.495 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep 'Watchers: none' 2024-04-24T14:35:14.523 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:14.523 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:14.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:35:14.562 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:14.562+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:14.575 INFO:tasks.workunit.client.0.smithi173.stdout:Watchers: none 2024-04-24T14:35:14.576 INFO:tasks.workunit.client.0.smithi173.stderr:+ break 2024-04-24T14:35:14.576 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd -p rbd --namespace '' snap purge testrbdnbd16809 2024-04-24T14:35:14.636 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:14.636 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:14.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:35:14.662 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:14.664+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:14.763 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:14.764+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:14.795 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:14.795 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:14.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:35:14.864 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:14.868+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:14.904 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:14.904 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:14.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:35:14.965 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:14.970+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:15.014 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:15.014 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:15.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:35:15.066 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:15.068+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:15.121 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:15.122 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:15.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:35:15.167 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:15.172+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:15.229 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:15.230 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:15.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:35:15.268 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:15.270+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:15.338 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:15.338 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:15.347 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:35:15.355 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5] out_osds: [1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 1] 2024-04-24T14:35:15.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T14:35:15.390 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:15.394+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:15.459 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:15.459 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:15.468 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:35:15.469 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:15.472+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:15.564 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:15.565 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:15.570 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:15.572+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:15.574 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:35:15.671 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:15.676+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:15.672 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:15.672 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:15.681 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:35:15.772 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:15.774+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:15.781 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:15.781 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:15.790 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:35:15.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:35:15.873 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:15.874+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:15.887 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:15.887 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:15.896 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:35:15.949 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:35:15.968 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:35:15.974 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:15.978+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:15.985 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:35:15.999 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:15.999 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:16.005 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:35:16.009 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:35:16.029 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:35:16.052 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:35:16.054 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:35:16.054 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:35:16.054 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:35:16.054 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:35:16.055 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:35:16.055 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:35:16.074 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:16.076+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:16.104 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:16.105 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:16.105 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:35:16.105 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:16.114 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:35:16.175 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:16.180+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:16.215 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:16.215 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:16.215 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:35:16.215 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:35:16.215 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:16.224 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:35:16.276 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:16.280+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:16.323 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:16.323 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:35:16.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:35:16.323 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:35:16.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.323 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 96.254723717999994, 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:16.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:35:16.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:16.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:35:16.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:35:16.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:35:16.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:35:16.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:35:16.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:35:16.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:35:16.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:35:16.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:35:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:35:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:35:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:35:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:35:16.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:35:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:35:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.259909139000001, 2024-04-24T14:35:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:35:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:16.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:16.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:35:16.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:16.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:35:16.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:35:16.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:35:16.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:35:16.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:35:16.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:35:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:35:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:35:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:35:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:35:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:16.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:35:16.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:35:16.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:35:16.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:35:16.332 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.332 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.332 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.332 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:35:16.332 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:16.332 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.248979880999997, 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:35:16.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:16.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:35:16.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:35:16.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:35:16.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:35:16.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:35:16.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:35:16.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:35:16.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:35:16.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:35:16.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:35:16.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:16.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:35:16.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:35:16.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:35:16.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:35:16.336 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.336 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.336 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.337 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:35:16.337 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:16.337 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.238023799999993, 2024-04-24T14:35:16.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:35:16.337 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.337 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.337 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:16.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:16.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:35:16.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:16.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:35:16.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:35:16.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:35:16.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:35:16.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:35:16.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:35:16.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:35:16.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:35:16.341 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.341 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.341 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.341 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:35:16.341 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:16.341 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.226454434999994, 2024-04-24T14:35:16.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:35:16.341 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.341 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.341 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:16.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:16.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:35:16.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:16.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:35:16.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:35:16.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:35:16.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:35:16.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:16.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:35:16.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:35:16.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:35:16.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:35:16.345 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.345 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.345 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.346 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:35:16.346 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:16.346 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.973334776000002, 2024-04-24T14:35:16.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:35:16.346 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.346 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.346 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:16.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:16.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:35:16.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:16.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:35:16.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:35:16.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:35:16.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:35:16.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:35:16.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:16.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:35:16.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:35:16.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:35:16.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:35:16.350 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.350 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.350 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.350 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:35:16.350 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:16.350 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.962589608000002, 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:35:16.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:16.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:35:16.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:35:16.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:35:16.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:35:16.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:35:16.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:35:16.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:35:16.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:35:16.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:35:16.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:35:16.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:16.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:35:16.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:35:16.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:35:16.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:35:16.354 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.354 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.354 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.355 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:35:16.355 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:16.355 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.950223331999993, 2024-04-24T14:35:16.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:35:16.355 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.355 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.355 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:16.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:35:16.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:35:16.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:35:16.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:35:16.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:35:16.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:35:16.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:35:16.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:35:16.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:16.358 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:35:16.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:35:16.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.358 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:35:16.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:35:16.359 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.359 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.359 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.359 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:35:16.359 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:16.359 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.796171525000005, 2024-04-24T14:35:16.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:35:16.359 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.359 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:35:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:16.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:35:16.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:35:16.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:35:16.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:35:16.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:35:16.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:35:16.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:35:16.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:35:16.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:35:16.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:35:16.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:16.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:35:16.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:35:16.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:35:16.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:35:16.363 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.363 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.363 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.363 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.626332848999994, 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:16.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:35:16.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:16.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:35:16.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:35:16.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:35:16.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:35:16.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:35:16.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:35:16.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:35:16.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:35:16.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:35:16.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:35:16.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:16.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:35:16.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:35:16.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:35:16.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:35:16.367 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.367 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.368 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.368 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:35:16.368 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:16.368 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 77.291362014000001, 2024-04-24T14:35:16.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:35:16.368 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.368 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.368 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:35:16.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:35:16.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:35:16.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:35:16.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:35:16.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:35:16.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:35:16.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:35:16.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:35:16.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:35:16.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:16.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:35:16.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:35:16.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 73.447211265999996, 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.372 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:16.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:35:16.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:35:16.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:35:16.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:35:16.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.375 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.375 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:35:16.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:35:16.375 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.375 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.375 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.376 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:35:16.376 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:16.376 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 58.223448265999998, 2024-04-24T14:35:16.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044084859999999997, 2024-04-24T14:35:16.376 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.376 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:16.376 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:16.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:16.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103206+0000", 2024-04-24T14:35:16.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:16.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104731+0000", 2024-04-24T14:35:16.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001524755 2024-04-24T14:35:16.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104759+0000", 2024-04-24T14:35:16.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7973000000000001e-05 2024-04-24T14:35:16.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104807+0000", 2024-04-24T14:35:16.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8112000000000003e-05 2024-04-24T14:35:16.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104907+0000", 2024-04-24T14:35:16.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100317 2024-04-24T14:35:16.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104918+0000", 2024-04-24T14:35:16.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747999999999999e-05 2024-04-24T14:35:16.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.107616+0000", 2024-04-24T14:35:16.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026977559999999999 2024-04-24T14:35:16.380 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.380 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.380 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.380 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:35:16.380 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:16.380 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 58.200153313000001, 2024-04-24T14:35:16.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046193859999999996, 2024-04-24T14:35:16.380 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.380 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:16.380 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:16.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:16.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126501+0000", 2024-04-24T14:35:16.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:16.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128169+0000", 2024-04-24T14:35:16.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016686229999999999 2024-04-24T14:35:16.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128202+0000", 2024-04-24T14:35:16.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2539000000000001e-05 2024-04-24T14:35:16.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128222+0000", 2024-04-24T14:35:16.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0149e-05 2024-04-24T14:35:16.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128375+0000", 2024-04-24T14:35:16.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000153459 2024-04-24T14:35:16.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128399+0000", 2024-04-24T14:35:16.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3393000000000001e-05 2024-04-24T14:35:16.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.131122+0000", 2024-04-24T14:35:16.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027227140000000002 2024-04-24T14:35:16.384 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.384 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.384 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.384 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5674.0:27 3.2 3:5592b7cf:::journal.12ff64619e2d:head [watch watch cookie 139908626846656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T14:35:16.384 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.218271+0000", 2024-04-24T14:35:16.385 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 12.10838472, 2024-04-24T14:35:16.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.107948372, 2024-04-24T14:35:16.385 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.385 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.385 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:16.385 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5674", 2024-04-24T14:35:16.385 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1498157813", 2024-04-24T14:35:16.385 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 27 2024-04-24T14:35:16.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.385 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218271+0000", 2024-04-24T14:35:16.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218271+0000", 2024-04-24T14:35:16.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218267+0000", 2024-04-24T14:35:16.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:35:16.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218280+0000", 2024-04-24T14:35:16.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2469e-05 2024-04-24T14:35:16.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218303+0000", 2024-04-24T14:35:16.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2952999999999999e-05 2024-04-24T14:35:16.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218316+0000", 2024-04-24T14:35:16.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.313e-05 2024-04-24T14:35:16.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218446+0000", 2024-04-24T14:35:16.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000130154 2024-04-24T14:35:16.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:35:16.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218460+0000", 2024-04-24T14:35:16.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3852999999999999e-05 2024-04-24T14:35:16.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.320512+0000", 2024-04-24T14:35:16.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.102051709 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for missing object", 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.320660+0000", 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000148362 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.322855+0000", 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021955249999999998 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.322976+0000", 2024-04-24T14:35:16.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000120813 2024-04-24T14:35:16.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.323702+0000", 2024-04-24T14:35:16.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00072575199999999997 2024-04-24T14:35:16.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.390 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.323827+0000", 2024-04-24T14:35:16.390 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012478200000000001 2024-04-24T14:35:16.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.390 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.390 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:16.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.324040+0000", 2024-04-24T14:35:16.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021278700000000001 2024-04-24T14:35:16.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:16.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.324936+0000", 2024-04-24T14:35:16.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00089639600000000004 2024-04-24T14:35:16.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.391 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.391 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:16.391 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.326067+0000", 2024-04-24T14:35:16.391 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011309110000000001 2024-04-24T14:35:16.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:16.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.326114+0000", 2024-04-24T14:35:16.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6903999999999997e-05 2024-04-24T14:35:16.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.392 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.392 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.392 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.326219+0000", 2024-04-24T14:35:16.392 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105368 2024-04-24T14:35:16.392 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.392 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 108/107 [PushOp(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 49'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@49'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:35:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.396664+0000", 2024-04-24T14:35:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.929991802, 2024-04-24T14:35:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004962241, 2024-04-24T14:35:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.393 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.396664+0000", 2024-04-24T14:35:16.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.396664+0000", 2024-04-24T14:35:16.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.394 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.396662+0000", 2024-04-24T14:35:16.394 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:16.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.394 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.394 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.398018+0000", 2024-04-24T14:35:16.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001355904 2024-04-24T14:35:16.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.398037+0000", 2024-04-24T14:35:16.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8907e-05 2024-04-24T14:35:16.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.395 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.395 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.398049+0000", 2024-04-24T14:35:16.395 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1661000000000001e-05 2024-04-24T14:35:16.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.395 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.400590+0000", 2024-04-24T14:35:16.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025413749999999998 2024-04-24T14:35:16.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.400599+0000", 2024-04-24T14:35:16.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0420000000000008e-06 2024-04-24T14:35:16.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.396 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.396 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.396 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.401626+0000", 2024-04-24T14:35:16.396 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0010267119999999999 2024-04-24T14:35:16.396 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.397 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.397 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.397 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:8 2.2 2:4f9137c5:::journal_data.2.12ff64619e2d.3:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e108)", 2024-04-24T14:35:16.397 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.439647+0000", 2024-04-24T14:35:16.397 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.887008162000001, 2024-04-24T14:35:16.397 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050746209999999996, 2024-04-24T14:35:16.397 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.397 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:16.397 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.397 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439647+0000", 2024-04-24T14:35:16.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439647+0000", 2024-04-24T14:35:16.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.398 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.398 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439646+0000", 2024-04-24T14:35:16.398 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:16.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.398 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439652+0000", 2024-04-24T14:35:16.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0170000000000004e-06 2024-04-24T14:35:16.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439660+0000", 2024-04-24T14:35:16.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7730000000000005e-06 2024-04-24T14:35:16.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.399 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.399 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.399 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439665+0000", 2024-04-24T14:35:16.399 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5220000000000001e-06 2024-04-24T14:35:16.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.441882+0000", 2024-04-24T14:35:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022169260000000001 2024-04-24T14:35:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.442038+0000", 2024-04-24T14:35:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000156661 2024-04-24T14:35:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.443062+0000", 2024-04-24T14:35:16.400 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001023604 2024-04-24T14:35:16.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.443281+0000", 2024-04-24T14:35:16.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000218564 2024-04-24T14:35:16.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.401 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.401 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.401 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.444722+0000", 2024-04-24T14:35:16.401 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0014413539999999999 2024-04-24T14:35:16.401 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.401 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.402 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.402 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.402 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5674.0:200 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 3895296~36864] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-04-24T14:35:16.402 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.470076+0000", 2024-04-24T14:35:16.402 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.856579801000001, 2024-04-24T14:35:16.402 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044721860000000004, 2024-04-24T14:35:16.402 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.402 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:16.402 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:16.402 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5674", 2024-04-24T14:35:16.402 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1498157813", 2024-04-24T14:35:16.402 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 200 2024-04-24T14:35:16.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.403 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470076+0000", 2024-04-24T14:35:16.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.403 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470076+0000", 2024-04-24T14:35:16.403 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.403 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.403 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470075+0000", 2024-04-24T14:35:16.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:16.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470078+0000", 2024-04-24T14:35:16.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5459999999999998e-06 2024-04-24T14:35:16.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.404 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.404 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.404 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470083+0000", 2024-04-24T14:35:16.404 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2009999999999998e-06 2024-04-24T14:35:16.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470088+0000", 2024-04-24T14:35:16.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7480000000000001e-06 2024-04-24T14:35:16.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473569+0000", 2024-04-24T14:35:16.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034816399999999998 2024-04-24T14:35:16.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.405 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.405 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.405 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473887+0000", 2024-04-24T14:35:16.405 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00031747200000000001 2024-04-24T14:35:16.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.406 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.406 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.474548+0000", 2024-04-24T14:35:16.406 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066107099999999995 2024-04-24T14:35:16.406 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.406 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.406 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.406 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.406 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 108/107 [PushOp(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 49'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@49'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:35:16.406 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.470456+0000", 2024-04-24T14:35:16.406 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.856199768, 2024-04-24T14:35:16.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055876789999999999, 2024-04-24T14:35:16.407 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.407 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:16.407 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470456+0000", 2024-04-24T14:35:16.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.407 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.407 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470456+0000", 2024-04-24T14:35:16.407 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470454+0000", 2024-04-24T14:35:16.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:16.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470992+0000", 2024-04-24T14:35:16.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053743599999999997 2024-04-24T14:35:16.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.408 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.408 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.408 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.471024+0000", 2024-04-24T14:35:16.408 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2480000000000001e-05 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.471045+0000", 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0503e-05 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473545+0000", 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025000299999999999 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473565+0000", 2024-04-24T14:35:16.409 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0352000000000001e-05 2024-04-24T14:35:16.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.410 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.410 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.476043+0000", 2024-04-24T14:35:16.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024783840000000001 2024-04-24T14:35:16.410 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.410 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.410 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.410 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.410 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5606.0:3 3.2 3:5592b7cf:::journal.12ff64619e2d:head [watch watch cookie 94231437332288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-24T14:35:16.410 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.527849+0000", 2024-04-24T14:35:16.410 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 11.79880621, 2024-04-24T14:35:16.410 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044396410000000002, 2024-04-24T14:35:16.411 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:16.411 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.411 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:16.411 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5606", 2024-04-24T14:35:16.411 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2386300588", 2024-04-24T14:35:16.411 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 3 2024-04-24T14:35:16.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.411 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:16.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.411 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:16.411 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527849+0000", 2024-04-24T14:35:16.411 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.411 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527849+0000", 2024-04-24T14:35:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527848+0000", 2024-04-24T14:35:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527855+0000", 2024-04-24T14:35:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8719999999999996e-06 2024-04-24T14:35:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:16.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527890+0000", 2024-04-24T14:35:16.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4836999999999999e-05 2024-04-24T14:35:16.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527906+0000", 2024-04-24T14:35:16.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6535999999999999e-05 2024-04-24T14:35:16.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.413 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.413 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:16.413 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.529465+0000", 2024-04-24T14:35:16.413 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001559429 2024-04-24T14:35:16.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:16.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.529800+0000", 2024-04-24T14:35:16.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000334638 2024-04-24T14:35:16.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:16.414 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.530413+0000", 2024-04-24T14:35:16.414 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061266800000000002 2024-04-24T14:35:16.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.414 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.414 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532044+0000", 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001631565 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532061+0000", 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6874000000000001e-05 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532139+0000", 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8191999999999999e-05 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:16.415 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:16.416 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532289+0000", 2024-04-24T14:35:16.416 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014948199999999999 2024-04-24T14:35:16.416 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.416 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.416 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.416 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:16.416 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:16.416 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:16.416 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T14:35:16.418 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T14:35:16.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:35:16.419 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:16.420+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:16.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T14:35:16.518 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:16.522+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:16.577 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:16.577 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:35:16.577 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:16.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:35:16.619 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:16.622+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:16.682 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:16.682 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:16.683 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:16.683 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:16.683 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:16.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:35:16.720 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:16.724+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:16.821 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:16.822+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:16.835 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:16.835 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:16.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:16.835 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:16.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.836 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:16.836 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:16.836 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 74.958925362000002, 2024-04-24T14:35:16.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:35:16.836 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.836 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:16.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:16.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:16.836 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:35:16.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.836 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:16.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:16.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:35:16.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:16.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:35:16.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:35:16.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:35:16.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:35:16.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:35:16.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:35:16.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:35:16.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:35:16.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:16.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:35:16.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:35:16.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:35:16.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:35:16.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:16.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:35:16.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:35:16.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:35:16.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:35:16.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:35:16.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:35:16.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:35:16.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:35:16.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.841 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.842 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:16.842 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:16.842 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 74.958896377000002, 2024-04-24T14:35:16.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:35:16.842 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.842 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:16.842 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.842 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:16.842 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:16.842 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:35:16.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.842 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:16.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:16.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:16.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:16.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:35:16.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:35:16.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:35:16.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:35:16.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:35:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:35:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:35:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:35:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:35:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:35:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:35:16.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:35:16.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:16.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:35:16.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:35:16.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:35:16.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:35:16.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:35:16.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:35:16.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:35:16.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:35:16.847 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.847 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.847 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.848 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:16.848 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:16.848 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 74.958877892000004, 2024-04-24T14:35:16.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:35:16.848 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.848 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:16.848 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.848 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:16.848 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:16.848 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:35:16.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.848 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:16.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:16.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:35:16.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:16.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:35:16.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:35:16.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:35:16.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:35:16.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:35:16.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:35:16.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:35:16.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:35:16.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:16.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:35:16.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:35:16.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:35:16.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:35:16.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:16.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:35:16.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:35:16.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:35:16.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:35:16.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:35:16.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:35:16.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:35:16.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:35:16.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.853 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.853 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:16.854 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:16.854 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 74.958862611000001, 2024-04-24T14:35:16.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:35:16.854 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.854 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:16.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:16.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:16.854 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:35:16.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.854 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:16.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:16.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:16.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:16.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:35:16.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:35:16.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:16.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:35:16.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:35:16.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:35:16.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:35:16.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:16.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:35:16.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:35:16.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:35:16.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:35:16.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:35:16.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:35:16.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:35:16.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:35:16.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.859 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.859 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:16.859 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:16.859 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 74.958846829999999, 2024-04-24T14:35:16.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:35:16.859 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.860 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:16.860 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.860 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:16.860 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:16.860 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:35:16.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.860 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:16.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:16.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:16.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:16.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:35:16.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:35:16.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:35:16.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:35:16.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:35:16.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:35:16.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:35:16.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:35:16.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:35:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:35:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:35:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:35:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:35:16.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:35:16.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:35:16.865 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.865 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.865 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.865 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:16.865 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:16.865 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 74.958832541999996, 2024-04-24T14:35:16.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:35:16.865 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.865 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:16.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:16.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:16.866 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:35:16.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.866 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:16.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:16.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:16.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:16.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:35:16.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:35:16.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:35:16.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:35:16.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:35:16.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:35:16.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:35:16.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:35:16.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:16.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:35:16.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:35:16.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:35:16.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:35:16.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:16.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:35:16.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:35:16.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:35:16.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:35:16.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:35:16.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:35:16.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:35:16.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:35:16.870 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.870 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.870 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.871 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:16.871 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:16.871 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 74.958812789999996, 2024-04-24T14:35:16.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:35:16.871 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.871 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:16.871 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.871 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:16.871 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:16.871 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:35:16.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.871 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:16.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:16.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:16.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:16.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:35:16.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:35:16.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:35:16.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:35:16.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:35:16.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:35:16.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:35:16.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:35:16.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:16.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:35:16.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:35:16.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:35:16.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:35:16.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:35:16.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:35:16.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.876 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.876 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:16.876 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:16.877 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 74.958795343000006, 2024-04-24T14:35:16.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:35:16.877 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.877 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:16.877 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.877 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:16.877 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:16.877 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:35:16.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.877 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:16.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:16.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:16.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:16.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:35:16.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:35:16.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:35:16.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:35:16.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:35:16.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:35:16.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:35:16.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:35:16.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:16.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:35:16.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:35:16.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:35:16.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:35:16.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:16.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:35:16.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:35:16.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:35:16.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:35:16.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:35:16.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:35:16.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:35:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:35:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:100 3.2 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.084740527999998, 2024-04-24T14:35:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009078015, 2024-04-24T14:35:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 100 2024-04-24T14:35:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754019+0000", 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754025+0000", 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2110000000000003e-06 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754037+0000", 2024-04-24T14:35:16.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1868999999999999e-05 2024-04-24T14:35:16.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754046+0000", 2024-04-24T14:35:16.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7430000000000008e-06 2024-04-24T14:35:16.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754095+0000", 2024-04-24T14:35:16.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8924000000000001e-05 2024-04-24T14:35:16.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754153+0000", 2024-04-24T14:35:16.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8165000000000003e-05 2024-04-24T14:35:16.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:16.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754262+0000", 2024-04-24T14:35:16.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108694 2024-04-24T14:35:16.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:16.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.762999+0000", 2024-04-24T14:35:16.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087372590000000007 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763027+0000", 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8467e-05 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763070+0000", 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2236e-05 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763098+0000", 2024-04-24T14:35:16.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8561999999999999e-05 2024-04-24T14:35:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:101 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [delete] snapc 7=[7,6,4] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.879532861999998, 2024-04-24T14:35:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012894743, 2024-04-24T14:35:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:16.888 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 101 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959227+0000", 2024-04-24T14:35:16.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:16.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959233+0000", 2024-04-24T14:35:16.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.857e-06 2024-04-24T14:35:16.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959249+0000", 2024-04-24T14:35:16.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6155999999999999e-05 2024-04-24T14:35:16.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959257+0000", 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7440000000000007e-06 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959790+0000", 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053381100000000003 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960206+0000", 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041549700000000002 2024-04-24T14:35:16.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:16.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960744+0000", 2024-04-24T14:35:16.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053762199999999997 2024-04-24T14:35:16.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:16.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971963+0000", 2024-04-24T14:35:16.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011219816000000001 2024-04-24T14:35:16.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:16.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972021+0000", 2024-04-24T14:35:16.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7855000000000001e-05 2024-04-24T14:35:16.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:16.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972073+0000", 2024-04-24T14:35:16.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2196e-05 2024-04-24T14:35:16.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972123+0000", 2024-04-24T14:35:16.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9311999999999998e-05 2024-04-24T14:35:16.893 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.893 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.893 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.894 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:102 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:16.894 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:16.894 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.879492835999997, 2024-04-24T14:35:16.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012939713, 2024-04-24T14:35:16.894 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.894 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.894 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.894 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:16.894 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:16.894 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 102 2024-04-24T14:35:16.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.894 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959267+0000", 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959274+0000", 2024-04-24T14:35:16.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.593e-06 2024-04-24T14:35:16.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959291+0000", 2024-04-24T14:35:16.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6387000000000001e-05 2024-04-24T14:35:16.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959301+0000", 2024-04-24T14:35:16.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0023e-05 2024-04-24T14:35:16.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969594+0000", 2024-04-24T14:35:16.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010293182 2024-04-24T14:35:16.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969902+0000", 2024-04-24T14:35:16.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030835299999999998 2024-04-24T14:35:16.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:16.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970408+0000", 2024-04-24T14:35:16.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050588100000000002 2024-04-24T14:35:16.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:16.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972009+0000", 2024-04-24T14:35:16.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016011210000000001 2024-04-24T14:35:16.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:16.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972132+0000", 2024-04-24T14:35:16.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122521 2024-04-24T14:35:16.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:16.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972174+0000", 2024-04-24T14:35:16.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2242e-05 2024-04-24T14:35:16.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972208+0000", 2024-04-24T14:35:16.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3522999999999997e-05 2024-04-24T14:35:16.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.899 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.899 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:104 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:16.899 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:16.899 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.878881622000002, 2024-04-24T14:35:16.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014589062, 2024-04-24T14:35:16.900 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.900 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.900 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.900 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:16.900 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:16.900 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 104 2024-04-24T14:35:16.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.900 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:16.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:16.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959877+0000", 2024-04-24T14:35:16.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:16.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959890+0000", 2024-04-24T14:35:16.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2537e-05 2024-04-24T14:35:16.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959909+0000", 2024-04-24T14:35:16.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9340000000000001e-05 2024-04-24T14:35:16.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959922+0000", 2024-04-24T14:35:16.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2882e-05 2024-04-24T14:35:16.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970900+0000", 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010977736 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971039+0000", 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139593 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971453+0000", 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041351099999999998 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:16.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974044+0000", 2024-04-24T14:35:16.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025908289999999998 2024-04-24T14:35:16.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:16.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974377+0000", 2024-04-24T14:35:16.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033310399999999997 2024-04-24T14:35:16.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:16.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974433+0000", 2024-04-24T14:35:16.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6166999999999999e-05 2024-04-24T14:35:16.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974468+0000", 2024-04-24T14:35:16.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5336000000000002e-05 2024-04-24T14:35:16.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.905 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.905 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:106 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:16.905 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:16.905 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.878822288999999, 2024-04-24T14:35:16.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017192064999999999, 2024-04-24T14:35:16.905 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.906 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.906 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.906 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:16.906 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:16.906 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 106 2024-04-24T14:35:16.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.906 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:16.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:16.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959938+0000", 2024-04-24T14:35:16.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:16.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959943+0000", 2024-04-24T14:35:16.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.677e-06 2024-04-24T14:35:16.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959953+0000", 2024-04-24T14:35:16.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8069999999999994e-06 2024-04-24T14:35:16.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:35:16.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0560000000000001e-06 2024-04-24T14:35:16.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972232+0000", 2024-04-24T14:35:16.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012272460000000001 2024-04-24T14:35:16.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972304+0000", 2024-04-24T14:35:16.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2630000000000001e-05 2024-04-24T14:35:16.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:16.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972548+0000", 2024-04-24T14:35:16.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000244047 2024-04-24T14:35:16.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976900+0000", 2024-04-24T14:35:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043514069999999998 2024-04-24T14:35:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977076+0000", 2024-04-24T14:35:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176281 2024-04-24T14:35:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977106+0000", 2024-04-24T14:35:16.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0015e-05 2024-04-24T14:35:16.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977131+0000", 2024-04-24T14:35:16.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4669e-05 2024-04-24T14:35:16.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.911 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.911 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:114 3.2 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:16.911 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:16.911 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.855501125000004, 2024-04-24T14:35:16.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090880089999999993, 2024-04-24T14:35:16.912 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.912 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:16.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:16.912 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 114 2024-04-24T14:35:16.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.912 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:16.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:16.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983259+0000", 2024-04-24T14:35:16.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:16.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983264+0000", 2024-04-24T14:35:16.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9130000000000002e-06 2024-04-24T14:35:16.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983277+0000", 2024-04-24T14:35:16.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3723999999999999e-05 2024-04-24T14:35:16.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983282+0000", 2024-04-24T14:35:16.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0429999999999998e-06 2024-04-24T14:35:16.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983339+0000", 2024-04-24T14:35:16.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6323000000000003e-05 2024-04-24T14:35:16.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983464+0000", 2024-04-24T14:35:16.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012554299999999999 2024-04-24T14:35:16.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:16.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983693+0000", 2024-04-24T14:35:16.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000228848 2024-04-24T14:35:16.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.986978+0000", 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032852810000000001 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992288+0000", 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053095369999999996 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992322+0000", 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3683999999999998e-05 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992348+0000", 2024-04-24T14:35:16.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6157999999999999e-05 2024-04-24T14:35:16.917 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.917 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.917 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.917 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:121 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:16.917 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:16.917 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.843445025999998, 2024-04-24T14:35:16.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098798159999999996, 2024-04-24T14:35:16.917 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.917 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.918 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.918 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:16.918 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:16.918 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 121 2024-04-24T14:35:16.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.918 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:16.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:16.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995315+0000", 2024-04-24T14:35:16.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:16.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995320+0000", 2024-04-24T14:35:16.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0239999999999997e-06 2024-04-24T14:35:16.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995328+0000", 2024-04-24T14:35:16.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7800000000000001e-06 2024-04-24T14:35:16.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995334+0000", 2024-04-24T14:35:16.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2330000000000003e-06 2024-04-24T14:35:16.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998810+0000", 2024-04-24T14:35:16.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034756090000000002 2024-04-24T14:35:16.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998870+0000", 2024-04-24T14:35:16.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0625999999999999e-05 2024-04-24T14:35:16.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:16.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.999091+0000", 2024-04-24T14:35:16.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000220631 2024-04-24T14:35:16.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:16.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.001389+0000", 2024-04-24T14:35:16.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022977549999999998 2024-04-24T14:35:16.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:16.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005097+0000", 2024-04-24T14:35:16.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037085439999999998 2024-04-24T14:35:16.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:16.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005154+0000", 2024-04-24T14:35:16.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6208000000000001e-05 2024-04-24T14:35:16.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005196+0000", 2024-04-24T14:35:16.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2052000000000003e-05 2024-04-24T14:35:16.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.923 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.923 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:127 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:16.923 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:16.923 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.833143509999999, 2024-04-24T14:35:16.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013414918, 2024-04-24T14:35:16.923 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.923 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.923 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.924 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:16.924 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:16.924 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2024-04-24T14:35:16.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.924 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:16.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:16.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005616+0000", 2024-04-24T14:35:16.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:16.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005622+0000", 2024-04-24T14:35:16.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2020000000000003e-06 2024-04-24T14:35:16.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005632+0000", 2024-04-24T14:35:16.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2024-04-24T14:35:16.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005637+0000", 2024-04-24T14:35:16.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7439999999999998e-06 2024-04-24T14:35:16.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005947+0000", 2024-04-24T14:35:16.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000310792 2024-04-24T14:35:16.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006059+0000", 2024-04-24T14:35:16.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011192399999999999 2024-04-24T14:35:16.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:16.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006503+0000", 2024-04-24T14:35:16.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000443394 2024-04-24T14:35:16.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:16.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018864+0000", 2024-04-24T14:35:16.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012361162 2024-04-24T14:35:16.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:16.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018877+0000", 2024-04-24T14:35:16.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3497e-05 2024-04-24T14:35:16.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:16.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018965+0000", 2024-04-24T14:35:16.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7476000000000001e-05 2024-04-24T14:35:16.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019032+0000", 2024-04-24T14:35:16.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7469000000000001e-05 2024-04-24T14:35:16.929 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.929 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.929 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.929 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:129 3.2 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:16.929 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:16.929 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.833003277000003, 2024-04-24T14:35:16.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010059754000000001, 2024-04-24T14:35:16.930 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.930 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.930 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.930 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:16.930 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:16.930 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 129 2024-04-24T14:35:16.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.930 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:16.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:16.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005757+0000", 2024-04-24T14:35:16.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:16.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005760+0000", 2024-04-24T14:35:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2430000000000001e-06 2024-04-24T14:35:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005769+0000", 2024-04-24T14:35:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4810000000000006e-06 2024-04-24T14:35:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005773+0000", 2024-04-24T14:35:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:35:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005840+0000", 2024-04-24T14:35:16.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6934999999999995e-05 2024-04-24T14:35:16.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005961+0000", 2024-04-24T14:35:16.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012153 2024-04-24T14:35:16.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:16.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006181+0000", 2024-04-24T14:35:16.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021947500000000001 2024-04-24T14:35:16.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:16.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015520+0000", 2024-04-24T14:35:16.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093393049999999991 2024-04-24T14:35:16.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:16.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015733+0000", 2024-04-24T14:35:16.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021355699999999999 2024-04-24T14:35:16.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:16.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015773+0000", 2024-04-24T14:35:16.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9554000000000001e-05 2024-04-24T14:35:16.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015817+0000", 2024-04-24T14:35:16.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4325e-05 2024-04-24T14:35:16.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.935 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.935 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:131 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:16.936 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:16.936 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.832725396000001, 2024-04-24T14:35:16.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014251976, 2024-04-24T14:35:16.936 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.936 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.936 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.936 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:16.936 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:16.936 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131 2024-04-24T14:35:16.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.936 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:16.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:16.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006034+0000", 2024-04-24T14:35:16.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:16.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006042+0000", 2024-04-24T14:35:16.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2970000000000004e-06 2024-04-24T14:35:16.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006061+0000", 2024-04-24T14:35:16.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8505999999999999e-05 2024-04-24T14:35:16.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006071+0000", 2024-04-24T14:35:16.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0409e-05 2024-04-24T14:35:16.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007899+0000", 2024-04-24T14:35:16.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018274789999999999 2024-04-24T14:35:16.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008281+0000", 2024-04-24T14:35:16.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038244700000000001 2024-04-24T14:35:16.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:16.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008528+0000", 2024-04-24T14:35:16.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024723900000000001 2024-04-24T14:35:16.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:16.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019070+0000", 2024-04-24T14:35:16.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010541867999999999 2024-04-24T14:35:16.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:16.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020099+0000", 2024-04-24T14:35:16.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00102852 2024-04-24T14:35:16.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:16.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020226+0000", 2024-04-24T14:35:16.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012744300000000001 2024-04-24T14:35:16.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020287+0000", 2024-04-24T14:35:16.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1253000000000005e-05 2024-04-24T14:35:16.941 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.941 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.941 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.941 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:132 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:16.941 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:16.942 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.832671543000004, 2024-04-24T14:35:16.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015917981000000001, 2024-04-24T14:35:16.942 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.942 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:16.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:16.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:16.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:16.942 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 132 2024-04-24T14:35:16.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.942 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:16.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:16.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:16.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:16.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006093+0000", 2024-04-24T14:35:16.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3819999999999997e-06 2024-04-24T14:35:16.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006103+0000", 2024-04-24T14:35:16.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0352e-05 2024-04-24T14:35:16.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006110+0000", 2024-04-24T14:35:16.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3910000000000003e-06 2024-04-24T14:35:16.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015586+0000", 2024-04-24T14:35:16.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094759119999999995 2024-04-24T14:35:16.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015730+0000", 2024-04-24T14:35:16.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014487199999999999 2024-04-24T14:35:16.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:16.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015886+0000", 2024-04-24T14:35:16.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015573799999999999 2024-04-24T14:35:16.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:16.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020306+0000", 2024-04-24T14:35:16.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044199399999999998 2024-04-24T14:35:16.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:16.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021887+0000", 2024-04-24T14:35:16.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015809400000000001 2024-04-24T14:35:16.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:16.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021942+0000", 2024-04-24T14:35:16.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5071000000000002e-05 2024-04-24T14:35:16.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.022007+0000", 2024-04-24T14:35:16.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5110000000000005e-05 2024-04-24T14:35:16.947 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.947 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.947 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.947 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.5324.0:127 3.0 e87/69 ondisk, result = 0)", 2024-04-24T14:35:16.947 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:16.947 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.830962475, 2024-04-24T14:35:16.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011245125, 2024-04-24T14:35:16.948 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:16.948 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:16.948 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:16.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:16.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:16.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:16.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:16.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:16.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:16.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:16.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:16.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:16.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007804+0000", 2024-04-24T14:35:16.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6610000000000003e-06 2024-04-24T14:35:16.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:16.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007817+0000", 2024-04-24T14:35:16.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3030999999999999e-05 2024-04-24T14:35:16.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:16.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007832+0000", 2024-04-24T14:35:16.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.47e-05 2024-04-24T14:35:16.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:16.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018843+0000", 2024-04-24T14:35:16.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011010647 2024-04-24T14:35:16.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:16.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018853+0000", 2024-04-24T14:35:16.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319000000000001e-05 2024-04-24T14:35:16.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:16.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:16.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:16.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019043+0000", 2024-04-24T14:35:16.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019055600000000001 2024-04-24T14:35:16.951 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.951 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.951 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.951 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:16.952 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:16.952 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:16.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:35:16.954 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:16.956+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:17.054 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:17.058+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:17.061 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:17.061 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:35:17.061 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:17.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:35:17.155 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:17.158+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:17.167 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:17.167 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:17.167 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:17.167 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:17.167 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:17.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:35:17.256 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:17.260+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:17.282 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:17.282 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:17.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:17.282 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:17.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.283 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.283 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:17.283 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.529486070000004, 2024-04-24T14:35:17.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:35:17.283 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.283 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.283 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.283 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.283 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.283 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:35:17.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.283 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:17.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:17.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:35:17.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:17.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:35:17.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:35:17.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:35:17.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:35:17.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:35:17.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:35:17.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:35:17.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:35:17.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:35:17.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:35:17.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:35:17.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:35:17.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:35:17.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:35:17.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:35:17.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:35:17.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:35:17.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:35:17.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:35:17.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:35:17.289 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.289 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.289 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.289 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.289 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:17.289 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.529435235999998, 2024-04-24T14:35:17.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:35:17.289 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.289 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.289 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.290 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.290 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.290 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:35:17.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.290 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:17.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:35:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:35:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:35:17.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:35:17.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:35:17.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:35:17.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:35:17.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:35:17.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:35:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:35:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:35:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:35:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:35:17.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:35:17.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:35:17.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:35:17.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:35:17.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:35:17.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:35:17.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:35:17.295 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.295 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.295 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.295 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.295 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:17.295 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.529398928999996, 2024-04-24T14:35:17.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:35:17.295 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.295 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.296 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.296 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.296 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.296 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:35:17.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.296 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:17.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:17.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:35:17.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:17.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:35:17.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:35:17.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:35:17.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:35:17.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:35:17.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:35:17.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:35:17.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:35:17.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:35:17.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:35:17.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:35:17.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:35:17.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:35:17.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:35:17.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:35:17.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:35:17.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:35:17.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:35:17.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:35:17.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:35:17.301 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.301 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.301 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.301 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.301 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:17.301 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.525987678999996, 2024-04-24T14:35:17.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:35:17.301 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.301 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.302 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.302 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.302 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.302 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:35:17.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.302 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:17.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:17.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:35:17.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:17.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:35:17.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:35:17.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:35:17.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:35:17.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:35:17.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:35:17.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:35:17.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:35:17.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:35:17.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:35:17.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:35:17.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:35:17.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:35:17.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:35:17.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:35:17.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:35:17.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:35:17.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:35:17.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:35:17.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:35:17.307 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.307 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.307 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.307 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.307 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:17.307 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.525926002000006, 2024-04-24T14:35:17.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:35:17.307 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.307 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.307 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.308 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.308 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.308 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:35:17.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.308 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:17.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:17.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:35:17.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:17.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:35:17.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:35:17.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:35:17.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:35:17.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:35:17.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:35:17.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:35:17.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:35:17.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:35:17.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:35:17.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:35:17.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:35:17.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:35:17.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:35:17.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:35:17.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:35:17.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:35:17.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:35:17.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:35:17.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:35:17.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.313 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.313 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.313 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:17.313 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.525891426000001, 2024-04-24T14:35:17.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:35:17.313 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.313 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.313 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.314 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:35:17.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.314 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:17.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:17.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:17.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:17.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:35:17.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:35:17.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:35:17.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:35:17.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:35:17.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:35:17.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:35:17.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:35:17.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:35:17.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:35:17.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:35:17.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:35:17.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:35:17.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:35:17.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:35:17.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:35:17.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:35:17.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:35:17.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:35:17.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:35:17.319 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.319 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.319 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.319 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.319 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:17.319 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.525862086999993, 2024-04-24T14:35:17.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:35:17.319 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.319 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.320 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.320 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.320 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.320 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:35:17.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.320 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:17.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:17.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:35:17.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:17.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:35:17.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:35:17.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:35:17.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:35:17.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:35:17.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:35:17.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:35:17.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:35:17.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:35:17.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:35:17.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:35:17.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:35:17.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:35:17.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:35:17.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:35:17.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:35:17.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:35:17.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:35:17.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:35:17.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:35:17.325 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.325 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.325 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.325 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.325 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:17.325 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.525785505000002, 2024-04-24T14:35:17.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:35:17.325 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.326 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.326 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.326 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.326 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.326 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:35:17.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.326 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:17.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:17.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:35:17.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:17.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:35:17.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:35:17.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:35:17.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:35:17.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:35:17.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:35:17.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:35:17.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:35:17.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:35:17.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:35:17.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:35:17.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:35:17.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:35:17.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:35:17.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:35:17.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:35:17.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:35:17.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:35:17.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:35:17.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:35:17.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.331 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.331 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.331 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:17.331 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.525737960000001, 2024-04-24T14:35:17.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:35:17.332 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.332 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.332 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.332 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.332 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.332 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:35:17.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.332 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:17.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:17.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:35:17.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:17.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:35:17.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:35:17.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:35:17.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:35:17.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:35:17.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:35:17.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:35:17.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:35:17.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:35:17.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:35:17.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:35:17.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:35:17.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:35:17.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:35:17.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:35:17.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:35:17.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:35:17.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:35:17.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:35:17.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:35:17.337 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.337 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.337 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.337 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.337 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:17.337 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.525697317999999, 2024-04-24T14:35:17.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:35:17.338 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.338 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.338 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.338 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.338 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.338 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:35:17.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.338 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:17.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:17.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:17.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:17.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:35:17.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:35:17.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:35:17.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:35:17.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:35:17.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:35:17.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:35:17.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:35:17.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:35:17.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:35:17.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:35:17.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:35:17.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:35:17.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:35:17.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:35:17.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:35:17.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:35:17.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:35:17.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:35:17.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:35:17.343 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.343 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.343 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.343 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.343 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:17.343 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.525680206999994, 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:17.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:35:17.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:17.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:35:17.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:35:17.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:35:17.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:35:17.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:35:17.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:35:17.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:35:17.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:35:17.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:35:17.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:35:17.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:35:17.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:35:17.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:35:17.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:35:17.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:35:17.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:35:17.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:35:17.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:35:17.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:35:17.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:35:17.349 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.349 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.349 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.349 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:103 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:17.349 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:17.349 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.326114093999998, 2024-04-24T14:35:17.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032646351999999997, 2024-04-24T14:35:17.349 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.350 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:17.350 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.350 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:17.350 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:17.350 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 103 2024-04-24T14:35:17.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.350 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:17.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:17.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959674+0000", 2024-04-24T14:35:17.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:17.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959688+0000", 2024-04-24T14:35:17.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2024-04-24T14:35:17.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959722+0000", 2024-04-24T14:35:17.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3936000000000002e-05 2024-04-24T14:35:17.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959735+0000", 2024-04-24T14:35:17.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2926e-05 2024-04-24T14:35:17.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959921+0000", 2024-04-24T14:35:17.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185601 2024-04-24T14:35:17.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960324+0000", 2024-04-24T14:35:17.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040356700000000002 2024-04-24T14:35:17.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:17.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960839+0000", 2024-04-24T14:35:17.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051410599999999996 2024-04-24T14:35:17.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:17.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.975634+0000", 2024-04-24T14:35:17.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014795770999999999 2024-04-24T14:35:17.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:17.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992113+0000", 2024-04-24T14:35:17.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016478982999999999 2024-04-24T14:35:17.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:17.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992228+0000", 2024-04-24T14:35:17.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114295 2024-04-24T14:35:17.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992323+0000", 2024-04-24T14:35:17.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5344000000000001e-05 2024-04-24T14:35:17.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.355 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.355 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:105 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:17.355 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:17.355 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.325864316000001, 2024-04-24T14:35:17.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032581757000000003, 2024-04-24T14:35:17.355 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.355 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:17.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:17.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:17.356 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105 2024-04-24T14:35:17.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.356 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:17.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:17.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959925+0000", 2024-04-24T14:35:17.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:17.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959937+0000", 2024-04-24T14:35:17.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2391999999999999e-05 2024-04-24T14:35:17.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:35:17.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2535999999999999e-05 2024-04-24T14:35:17.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959970+0000", 2024-04-24T14:35:17.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1039999999999999e-05 2024-04-24T14:35:17.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969531+0000", 2024-04-24T14:35:17.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095603949999999993 2024-04-24T14:35:17.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969669+0000", 2024-04-24T14:35:17.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013806499999999999 2024-04-24T14:35:17.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:17.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970010+0000", 2024-04-24T14:35:17.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034062200000000001 2024-04-24T14:35:17.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:17.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976983+0000", 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069735600000000002 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992413+0000", 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015429841999999999 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992457+0000", 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4169999999999999e-05 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992508+0000", 2024-04-24T14:35:17.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0995000000000002e-05 2024-04-24T14:35:17.361 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.361 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.361 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.361 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:113 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:17.361 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:17.361 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.305638025, 2024-04-24T14:35:17.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014793863000000001, 2024-04-24T14:35:17.361 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.361 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:17.361 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.361 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:17.362 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:17.362 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 113 2024-04-24T14:35:17.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.362 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:17.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:17.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980152+0000", 2024-04-24T14:35:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980156+0000", 2024-04-24T14:35:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.619e-06 2024-04-24T14:35:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980163+0000", 2024-04-24T14:35:17.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7070000000000003e-06 2024-04-24T14:35:17.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980173+0000", 2024-04-24T14:35:17.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0957e-05 2024-04-24T14:35:17.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983381+0000", 2024-04-24T14:35:17.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032079930000000001 2024-04-24T14:35:17.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983628+0000", 2024-04-24T14:35:17.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024665700000000001 2024-04-24T14:35:17.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:17.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983964+0000", 2024-04-24T14:35:17.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000336332 2024-04-24T14:35:17.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:17.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992694+0000", 2024-04-24T14:35:17.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087297009999999994 2024-04-24T14:35:17.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:17.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994878+0000", 2024-04-24T14:35:17.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021841439999999998 2024-04-24T14:35:17.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:17.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994917+0000", 2024-04-24T14:35:17.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8544999999999998e-05 2024-04-24T14:35:17.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994947+0000", 2024-04-24T14:35:17.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9705000000000001e-05 2024-04-24T14:35:17.367 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.367 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.367 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.367 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:130 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:17.367 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:17.367 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.279944589999999, 2024-04-24T14:35:17.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014646289999999999, 2024-04-24T14:35:17.367 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.367 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:17.368 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.368 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:17.368 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:17.368 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 130 2024-04-24T14:35:17.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.368 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:17.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005844+0000", 2024-04-24T14:35:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005853+0000", 2024-04-24T14:35:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1319999999999994e-06 2024-04-24T14:35:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005870+0000", 2024-04-24T14:35:17.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7448999999999998e-05 2024-04-24T14:35:17.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005876+0000", 2024-04-24T14:35:17.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3319999999999997e-06 2024-04-24T14:35:17.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005913+0000", 2024-04-24T14:35:17.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6835999999999998e-05 2024-04-24T14:35:17.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006017+0000", 2024-04-24T14:35:17.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010358500000000001 2024-04-24T14:35:17.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:17.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006139+0000", 2024-04-24T14:35:17.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001227 2024-04-24T14:35:17.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:17.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.010194+0000", 2024-04-24T14:35:17.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040549380000000001 2024-04-24T14:35:17.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:17.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020365+0000", 2024-04-24T14:35:17.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010170983 2024-04-24T14:35:17.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:17.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020428+0000", 2024-04-24T14:35:17.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2457000000000002e-05 2024-04-24T14:35:17.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020492+0000", 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4592999999999996e-05 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:155 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1601536~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.246283802999997, 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015144391, 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.373 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:17.374 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:17.374 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 155 2024-04-24T14:35:17.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.374 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:17.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:17.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039506+0000", 2024-04-24T14:35:17.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:17.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039766+0000", 2024-04-24T14:35:17.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025994400000000002 2024-04-24T14:35:17.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039780+0000", 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3366e-05 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039787+0000", 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4359999999999996e-06 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.050886+0000", 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011098993 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051026+0000", 2024-04-24T14:35:17.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001395 2024-04-24T14:35:17.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:17.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051211+0000", 2024-04-24T14:35:17.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018547099999999999 2024-04-24T14:35:17.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:17.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054490+0000", 2024-04-24T14:35:17.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032785499999999999 2024-04-24T14:35:17.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:17.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054505+0000", 2024-04-24T14:35:17.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5906999999999999e-05 2024-04-24T14:35:17.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:17.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054594+0000", 2024-04-24T14:35:17.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8817e-05 2024-04-24T14:35:17.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054651+0000", 2024-04-24T14:35:17.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7040999999999998e-05 2024-04-24T14:35:17.379 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.379 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.379 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.379 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:156 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1732608~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:17.379 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:17.379 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.245993097000003, 2024-04-24T14:35:17.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016895194999999998, 2024-04-24T14:35:17.379 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.379 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:17.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:17.380 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:17.380 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 156 2024-04-24T14:35:17.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.380 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:17.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:17.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039797+0000", 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040065+0000", 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000268361 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040086+0000", 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1024000000000001e-05 2024-04-24T14:35:17.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040094+0000", 2024-04-24T14:35:17.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9379999999999992e-06 2024-04-24T14:35:17.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051936+0000", 2024-04-24T14:35:17.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011842518 2024-04-24T14:35:17.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052062+0000", 2024-04-24T14:35:17.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125482 2024-04-24T14:35:17.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:17.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052206+0000", 2024-04-24T14:35:17.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143953 2024-04-24T14:35:17.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:17.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054715+0000", 2024-04-24T14:35:17.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025088240000000002 2024-04-24T14:35:17.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:17.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056534+0000", 2024-04-24T14:35:17.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001819436 2024-04-24T14:35:17.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:17.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056610+0000", 2024-04-24T14:35:17.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5383000000000007e-05 2024-04-24T14:35:17.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056693+0000", 2024-04-24T14:35:17.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3283999999999997e-05 2024-04-24T14:35:17.384 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.384 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.385 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.385 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:157 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1863680~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:17.385 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:17.385 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.245685717999997, 2024-04-24T14:35:17.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016763674999999999, 2024-04-24T14:35:17.385 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.385 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:17.385 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.385 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:17.385 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:17.385 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157 2024-04-24T14:35:17.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.386 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:17.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:17.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040104+0000", 2024-04-24T14:35:17.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:17.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040244+0000", 2024-04-24T14:35:17.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014035499999999999 2024-04-24T14:35:17.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040255+0000", 2024-04-24T14:35:17.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0135999999999999e-05 2024-04-24T14:35:17.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040260+0000", 2024-04-24T14:35:17.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7919999999999998e-06 2024-04-24T14:35:17.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053406+0000", 2024-04-24T14:35:17.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013146023999999999 2024-04-24T14:35:17.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053531+0000", 2024-04-24T14:35:17.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012471800000000001 2024-04-24T14:35:17.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:17.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053715+0000", 2024-04-24T14:35:17.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018395099999999999 2024-04-24T14:35:17.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:17.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056711+0000", 2024-04-24T14:35:17.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029956520000000001 2024-04-24T14:35:17.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056783+0000", 2024-04-24T14:35:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2010999999999999e-05 2024-04-24T14:35:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056831+0000", 2024-04-24T14:35:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8189999999999998e-05 2024-04-24T14:35:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056869+0000", 2024-04-24T14:35:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7812000000000003e-05 2024-04-24T14:35:17.390 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:158 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1994752~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.245523790999997, 2024-04-24T14:35:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016845334999999999, 2024-04-24T14:35:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:17.391 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2024-04-24T14:35:17.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.392 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:17.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:17.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040266+0000", 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040379+0000", 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112728 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040387+0000", 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8499999999999994e-06 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040391+0000", 2024-04-24T14:35:17.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2719999999999999e-06 2024-04-24T14:35:17.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054803+0000", 2024-04-24T14:35:17.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014411548 2024-04-24T14:35:17.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054982+0000", 2024-04-24T14:35:17.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017941699999999999 2024-04-24T14:35:17.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:17.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.055167+0000", 2024-04-24T14:35:17.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018542100000000001 2024-04-24T14:35:17.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:17.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056718+0000", 2024-04-24T14:35:17.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015505670000000001 2024-04-24T14:35:17.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:17.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057020+0000", 2024-04-24T14:35:17.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030227500000000002 2024-04-24T14:35:17.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:17.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057070+0000", 2024-04-24T14:35:17.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9611000000000003e-05 2024-04-24T14:35:17.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057112+0000", 2024-04-24T14:35:17.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2373999999999998e-05 2024-04-24T14:35:17.396 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.396 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.397 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.397 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:159 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:17.397 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:17.397 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.240088802999999, 2024-04-24T14:35:17.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015930650000000001, 2024-04-24T14:35:17.397 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.397 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.397 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.397 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:17.397 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:17.397 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2024-04-24T14:35:17.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.398 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:17.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:17.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045701+0000", 2024-04-24T14:35:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045707+0000", 2024-04-24T14:35:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.562e-06 2024-04-24T14:35:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045728+0000", 2024-04-24T14:35:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1109999999999999e-05 2024-04-24T14:35:17.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045738+0000", 2024-04-24T14:35:17.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0016e-05 2024-04-24T14:35:17.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056183+0000", 2024-04-24T14:35:17.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010444836000000001 2024-04-24T14:35:17.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056470+0000", 2024-04-24T14:35:17.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028696899999999998 2024-04-24T14:35:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057160+0000", 2024-04-24T14:35:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069004699999999995 2024-04-24T14:35:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057291+0000", 2024-04-24T14:35:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013096200000000001 2024-04-24T14:35:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.061633+0000", 2024-04-24T14:35:17.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004341414 2024-04-24T14:35:17.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.402 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.402 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.402 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:17.403 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:17.408+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:17.404 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:35:17.404 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-24T14:35:17.405 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T14:35:17.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T14:35:17.499 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:17.499 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:17.499 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:35:17.499 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:17.504 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:17.506+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:17.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:35:17.605 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:17.606+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:17.617 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:17.617 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:17.617 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:17.617 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:17.617 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:17.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:35:17.706 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:17.708+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:17.771 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:17.771 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:17.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:17.771 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 96.303969960000003, 2024-04-24T14:35:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:35:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:35:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.772 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:35:17.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:17.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:35:17.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:35:17.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:35:17.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:35:17.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:35:17.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:35:17.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.776 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.776 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.776 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:17.776 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.928613256999995, 2024-04-24T14:35:17.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:35:17.776 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.776 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.777 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.777 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.777 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:35:17.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.777 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:17.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:17.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:35:17.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:17.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:35:17.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:35:17.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:35:17.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:35:17.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:35:17.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:35:17.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:35:17.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:35:17.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:17.780 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.927185102999999, 2024-04-24T14:35:17.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:35:17.781 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.781 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.781 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:35:17.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.781 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:17.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:17.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:35:17.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:17.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:35:17.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:35:17.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:35:17.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:35:17.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:35:17.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:35:17.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.784 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.927158014, 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:17.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:17.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:35:17.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:35:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:35:17.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:35:17.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:35:17.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:35:17.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:35:17.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:35:17.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:35:17.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.789 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.789 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.789 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:17.789 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.927134046999996, 2024-04-24T14:35:17.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:35:17.789 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.789 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.789 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.789 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.790 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:35:17.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.790 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:17.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:17.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:35:17.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:35:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:35:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:35:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:35:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:35:17.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.913150067999993, 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:35:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.794 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.794 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:35:17.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.794 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:17.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:35:17.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:35:17.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:35:17.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:35:17.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:35:17.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:35:17.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:35:17.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:35:17.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:35:17.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:35:17.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:35:17.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:35:17.798 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.913033756999994, 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.799 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:35:17.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:35:17.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:35:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:35:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:35:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:35:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:35:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:35:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:35:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:35:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:35:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:17.804 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.912986098999994, 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:35:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:35:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:35:17.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:35:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:35:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:35:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:35:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:35:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:35:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:35:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:35:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:35:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:35:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:35:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:35:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.908948800999994, 2024-04-24T14:35:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:35:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.810 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:35:17.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:35:17.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:35:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:35:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:35:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:35:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:35:17.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.815 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.908885041999994, 2024-04-24T14:35:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:35:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:35:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:35:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:35:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:35:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:35:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:35:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:35:17.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:35:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:35:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:35:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:35:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:35:17.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:35:17.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:35:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:35:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:35:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.908859551999996, 2024-04-24T14:35:17.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:35:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:35:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:35:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:35:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:35:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:35:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:35:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:35:17.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.908835801999999, 2024-04-24T14:35:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:35:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.827 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:35:17.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:35:17.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:35:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:35:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:35:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:35:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:35:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:35:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:35:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:35:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:35:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:17.832 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.908797289000006, 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:35:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:35:17.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:35:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:35:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:35:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:35:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:35:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:35:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:35:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:35:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:35:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:35:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:35:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:35:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:35:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:35:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:35:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:35:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:35:17.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:35:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:35:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.908769092, 2024-04-24T14:35:17.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:35:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:35:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:35:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:35:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:35:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:35:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:35:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:35:17.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:35:17.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:35:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:35:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:35:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:35:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:35:17.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.908747634999997, 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:35:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:35:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:35:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:35:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:35:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:35:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:35:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:35:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:35:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:35:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:35:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.849 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.908728792999995, 2024-04-24T14:35:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:35:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:35:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:35:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:35:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:35:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:35:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:35:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:35:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:35:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:35:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:35:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:35:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:35:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:35:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:35:17.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:35:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:35:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:35:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:35:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:35:17.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 78.695896644000001, 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:35:17.855 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:35:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:35:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:35:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:35:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:35:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:35:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:35:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:35:17.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:35:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:35:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:35:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:35:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:35:17.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:35:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:35:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:35:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:35:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:35:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:39 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1753088~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.756124227999997, 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01139394, 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:17.861 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 39 2024-04-24T14:35:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018967+0000", 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019242+0000", 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027504500000000002 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019274+0000", 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2472999999999999e-05 2024-04-24T14:35:17.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019288+0000", 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3283e-05 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026753+0000", 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074651120000000003 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026971+0000", 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021810800000000001 2024-04-24T14:35:17.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.027211+0000", 2024-04-24T14:35:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023975 2024-04-24T14:35:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030200+0000", 2024-04-24T14:35:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029890709999999998 2024-04-24T14:35:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:17.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030252+0000", 2024-04-24T14:35:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1926999999999999e-05 2024-04-24T14:35:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030312+0000", 2024-04-24T14:35:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0602e-05 2024-04-24T14:35:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030363+0000", 2024-04-24T14:35:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0342e-05 2024-04-24T14:35:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.866 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:40 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1884160~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.755788254000002, 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011199064, 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 40 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019303+0000", 2024-04-24T14:35:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:17.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019729+0000", 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042620899999999998 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019769+0000", 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9847e-05 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019797+0000", 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8347000000000002e-05 2024-04-24T14:35:17.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028299+0000", 2024-04-24T14:35:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085010629999999997 2024-04-24T14:35:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028385+0000", 2024-04-24T14:35:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6824999999999996e-05 2024-04-24T14:35:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028510+0000", 2024-04-24T14:35:17.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012483000000000001 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030410+0000", 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001899612 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030416+0000", 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7699999999999998e-06 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030461+0000", 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5157999999999997e-05 2024-04-24T14:35:17.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030504+0000", 2024-04-24T14:35:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2988999999999998e-05 2024-04-24T14:35:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:41 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 2015232~81920] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.755279547000001, 2024-04-24T14:35:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012345257, 2024-04-24T14:35:17.872 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 41 2024-04-24T14:35:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019812+0000", 2024-04-24T14:35:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019975+0000", 2024-04-24T14:35:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016325700000000001 2024-04-24T14:35:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020025+0000", 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9484999999999998e-05 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020035+0000", 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0444e-05 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029150+0000", 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091150160000000001 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029352+0000", 2024-04-24T14:35:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020166499999999999 2024-04-24T14:35:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029611+0000", 2024-04-24T14:35:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025949600000000001 2024-04-24T14:35:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030518+0000", 2024-04-24T14:35:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090633300000000005 2024-04-24T14:35:17.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032088+0000", 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015703290000000001 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032130+0000", 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2273e-05 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032159+0000", 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8387000000000001e-05 2024-04-24T14:35:17.877 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:17.878 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:17.878 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:17.879 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:35:17.880 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:17.884+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:17.980 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:17.986+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:17.987 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:17.987 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:17.987 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:35:17.987 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:17.996 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:35:18.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-24T14:35:18.098 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:18.098 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:18.098 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:35:18.098 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:35:18.098 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:18.107 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:35:18.118 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:18.122+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:18.181 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:18.184+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:18.209 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:35:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:35:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:35:18.209 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 175.588301137, 2024-04-24T14:35:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:35:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:35:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.210 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.211 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.212 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:35:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:35:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:35:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:35:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:35:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:35:18.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:35:18.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:35:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:35:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 175.58812405, 2024-04-24T14:35:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:35:18.215 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.216 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.217 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:35:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:35:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:35:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:35:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:35:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:35:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.218 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:35:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:35:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:35:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:35:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:35:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:35:18.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:35:18.220 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 175.58802013600001, 2024-04-24T14:35:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:35:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:18.221 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:35:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.222 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:35:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:35:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:35:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:35:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:35:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.223 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:35:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:35:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:35:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:35:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:35:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:35:18.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:35:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:35:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:35:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:35:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:35:18.225 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:35:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:35:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:35:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:35:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:35:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 175.58763168499999, 2024-04-24T14:35:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:35:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:35:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.227 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:35:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:35:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:35:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:35:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:35:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:35:18.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:35:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:35:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:35:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:35:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:35:18.230 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:35:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:35:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:35:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:35:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:35:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:18.231 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 175.587546473, 2024-04-24T14:35:18.232 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:35:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:35:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.233 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:35:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:35:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:35:18.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:35:18.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.236 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.236 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.236 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:35:18.236 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:35:18.236 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.236 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.236 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:18.236 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:35:18.236 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:35:18.236 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.236 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.236 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:18.236 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:35:18.237 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:35:18.237 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.237 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.237 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:18.237 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:35:18.237 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:35:18.237 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.237 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.237 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:18.237 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:35:18.237 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:35:18.237 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.237 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.237 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:35:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:35:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 175.587309592, 2024-04-24T14:35:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:35:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:18.238 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.240 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.240 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:35:18.240 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:18.240 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.240 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.240 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:35:18.240 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:35:18.240 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.240 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.240 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.240 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:35:18.240 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:35:18.241 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:35:18.242 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.242 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.242 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:18.242 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:35:18.242 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:35:18.242 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.242 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.242 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:18.242 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:35:18.242 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:35:18.242 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.242 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.242 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.243 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.244 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.244 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.244 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:18.244 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:18.244 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 175.58722807000001, 2024-04-24T14:35:18.244 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:35:18.244 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.244 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:18.244 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.244 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:18.244 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:18.244 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:35:18.244 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.244 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.245 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.245 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.245 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:18.245 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.245 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.245 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.245 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:18.245 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.245 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.245 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.245 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:35:18.245 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:18.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.246 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.246 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:35:18.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:35:18.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.246 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.246 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:35:18.246 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:35:18.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.246 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.246 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.246 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:18.247 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:18.248 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:35:18.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:35:18.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.249 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.249 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:35:18.249 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:35:18.249 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.249 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.249 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.249 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.249 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:18.249 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:18.249 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 175.155315723, 2024-04-24T14:35:18.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:35:18.250 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.250 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:18.250 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.250 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:18.250 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:18.250 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:35:18.250 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.250 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.250 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.250 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.250 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:18.250 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:35:18.251 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:35:18.252 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:35:18.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.253 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.253 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:35:18.253 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:35:18.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.253 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:18.253 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:35:18.253 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:35:18.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.253 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:18.253 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:35:18.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:35:18.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:18.254 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:35:18.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:35:18.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:18.254 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:35:18.254 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:35:18.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.254 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.254 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.255 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:35:18.255 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:35:18.255 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.255 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.255 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.255 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.255 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.255 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:18.255 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:18.255 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 175.155287423, 2024-04-24T14:35:18.255 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:35:18.255 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.255 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:18.255 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.256 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:18.256 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:18.256 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:35:18.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.256 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:18.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.256 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.256 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.256 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:18.256 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.257 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.257 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:35:18.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:18.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.257 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.257 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:35:18.257 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:35:18.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.257 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.257 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.257 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:35:18.258 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:35:18.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.258 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.258 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:35:18.258 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:35:18.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.258 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.258 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:35:18.258 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:35:18.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.258 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.258 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:35:18.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:35:18.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:18.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:35:18.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:35:18.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.259 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:18.259 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:35:18.259 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:35:18.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.259 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:18.260 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:35:18.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:35:18.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:18.260 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:35:18.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:35:18.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.260 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.260 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.260 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:35:18.260 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:35:18.260 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.261 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.261 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.261 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.261 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:18.261 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:18.261 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 175.155199719, 2024-04-24T14:35:18.261 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:35:18.261 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.261 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:18.261 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.261 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:18.261 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:18.261 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.262 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:35:18.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:18.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.263 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.263 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:35:18.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:35:18.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.263 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.263 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:35:18.263 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:35:18.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.263 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.263 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:35:18.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:35:18.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:35:18.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:35:18.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.264 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.264 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:35:18.264 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:35:18.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.264 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.265 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:18.266 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:35:18.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:35:18.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.266 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.266 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:35:18.266 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:35:18.266 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.266 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.266 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.266 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.266 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:18.266 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:18.267 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 175.15507691799999, 2024-04-24T14:35:18.267 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:35:18.267 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.267 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:18.267 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.267 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:18.267 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:18.267 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:35:18.267 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.267 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.267 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.267 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.267 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:18.267 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:35:18.268 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:35:18.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:35:18.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:35:18.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:35:18.269 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:35:18.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.269 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.269 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.269 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:35:18.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:35:18.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:35:18.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:35:18.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.270 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:18.270 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:35:18.270 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:35:18.270 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.270 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.271 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:35:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:35:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:35:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 105.848832358, 2024-04-24T14:35:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:35:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.272 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:18.273 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:35:18.274 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:18.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:35:18.274 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:35:18.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.274 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.274 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.274 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:35:18.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:35:18.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:35:18.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:35:18.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.275 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:35:18.275 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:35:18.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.275 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.275 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.276 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.277 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:35:18.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:35:18.277 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.277 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.277 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.277 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.277 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:18.277 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:18.277 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 98.154708358999997, 2024-04-24T14:35:18.277 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:35:18.277 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.277 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:18.277 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.278 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:35:18.278 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:35:18.278 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:35:18.278 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.278 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.278 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.278 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.278 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:18.278 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.278 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.278 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.278 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.278 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:18.278 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:35:18.279 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:35:18.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:35:18.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:35:18.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:35:18.280 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:35:18.280 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.280 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.280 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:18.280 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.281 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.282 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 79.195348393000003, 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.283 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.284 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:18.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.284 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.284 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:18.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.284 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.284 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:35:18.284 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:18.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.284 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.285 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.286 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:18.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:35:18.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:35:18.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:35:18.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:35:18.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.287 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.287 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.287 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:35:18.287 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 79.195267838000007, 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:35:18.288 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.289 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:18.289 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.289 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:18.289 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:18.289 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:35:18.289 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.289 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.289 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.289 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:18.289 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.289 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.289 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.289 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.290 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:18.290 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.290 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.290 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.290 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:35:18.290 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:18.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.290 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.290 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.290 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:35:18.290 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:35:18.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.291 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.291 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:35:18.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:35:18.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.291 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.291 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:35:18.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:35:18.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.291 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.291 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.291 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:35:18.291 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:35:18.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:18.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:35:18.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:35:18.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:35:18.292 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:35:18.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.292 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.292 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:18.292 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.293 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:35:18.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:35:18.294 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.294 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.294 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.294 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.294 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:18.294 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:18.294 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 79.195220109000005, 2024-04-24T14:35:18.294 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:35:18.294 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.294 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:18.294 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.295 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:18.295 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:18.295 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:35:18.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.295 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:18.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.295 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.295 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.295 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:18.295 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.296 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.296 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:35:18.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:18.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.296 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.296 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:35:18.296 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:35:18.296 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.296 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.296 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.296 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:35:18.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:35:18.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:35:18.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:35:18.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.297 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:35:18.297 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:35:18.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.297 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:18.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:35:18.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:35:18.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:35:18.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:35:18.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.298 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:18.298 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:35:18.298 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:35:18.298 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.298 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.299 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.300 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.300 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.300 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:18.300 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:18.300 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 79.195192345999999, 2024-04-24T14:35:18.300 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:35:18.300 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.300 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:18.300 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.300 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:18.300 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:18.300 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:35:18.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:35:18.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:18.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:35:18.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:35:18.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:35:18.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:35:18.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 79.195152910999994, 2024-04-24T14:35:18.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:35:18.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:35:18.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:35:18.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:35:18.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:35:18.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:35:18.310 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.310 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 79.195127927000001, 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.311 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:18.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:18.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:35:18.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:35:18.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:35:18.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:35:18.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 79.195102267999999, 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:35:18.316 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:18.317 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:18.317 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:18.317 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:18.317 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:18.317 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:35:18.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.317 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:18.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:18.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:18.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:35:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:35:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:35:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:18.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:35:18.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:35:18.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:35:18.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:35:18.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:18.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:35:18.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:35:18.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.321 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.322 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.322 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:18.322 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:18.322 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:18.324 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:35:18.324 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:35:18.324 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:35:18.324 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:35:18.324 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:35:18.324 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:35:18.325 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:35:18.325 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:18.328+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:18.325 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:35:18.325 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:35:18.325 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:35:18.325 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:35:18.325 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:35:18.325 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:35:18.402 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T14:35:18.409 INFO:tasks.workunit.client.0.smithi173.stderr: Removing all snapshots: 25% complete... Removing all snapshots: 50% complete... Removing all snapshots: 75% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-04-24T14:35:18.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-24T14:35:18.420 INFO:tasks.workunit.client.0.smithi173.stderr:2024-04-24T14:35:18.424+0000 7f167b3ae700 0 -- 172.21.15.173:0/1567306440 >> [v2:172.21.15.103:6808/20318,v1:172.21.15.103:6809/20318] conn(0x7f1660008980 msgr2=0x7f166000ae00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T14:35:18.423 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:18.428+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:18.427 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd -p rbd --namespace '' remove testrbdnbd16809 2024-04-24T14:35:18.436 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:18.436 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:18.446 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:35:18.524 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:18.528+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:18.601 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:18.601 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:18.612 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:35:18.614 INFO:tasks.workunit.client.0.smithi173.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-04-24T14:35:18.620+0000 7ff4eb5cd700 0 -- 172.21.15.173:0/2485957160 >> [v2:172.21.15.90:6808/20310,v1:172.21.15.90:6809/20310] conn(0x55665cb23f30 msgr2=0x55665cb26310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T14:35:18.624 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:18.626+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:18.642 INFO:tasks.workunit.client.0.smithi173.stderr: Removing image: 100% complete...done. 2024-04-24T14:35:18.648 INFO:tasks.workunit.client.0.smithi173.stderr:+ for ns in '' ${NS} 2024-04-24T14:35:18.649 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd -p rbd --namespace ns status testrbdnbd16809 2024-04-24T14:35:18.709 INFO:tasks.workunit.client.0.smithi173.stdout:Watchers: none 2024-04-24T14:35:18.712 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:18.712 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:18.715 INFO:tasks.workunit.client.0.smithi173.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-04-24T14:35:18.715 INFO:tasks.workunit.client.0.smithi173.stderr:+ sleep 0.5 2024-04-24T14:35:18.721 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:35:18.725 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:18.730+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:18.814 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:18.814 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:18.823 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:35:18.826 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:18.828+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:18.838 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.4 to 1 (10000) 2024-04-24T14:35:18.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-24T14:35:18.923 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:18.923 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:18.927 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:18.932+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:18.932 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:35:19.028 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:19.030+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:19.079 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:19.080 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:19.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:35:19.134 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:19.138+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:19.190 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:19.191 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:19.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:35:19.217 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd -p rbd --namespace ns status testrbdnbd16809 2024-04-24T14:35:19.217 INFO:tasks.workunit.client.0.smithi173.stderr:+ grep 'Watchers: none' 2024-04-24T14:35:19.230 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:19.234+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:19.282 INFO:tasks.workunit.client.0.smithi173.stdout:Watchers: none 2024-04-24T14:35:19.283 INFO:tasks.workunit.client.0.smithi173.stderr:+ break 2024-04-24T14:35:19.283 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd -p rbd --namespace ns snap purge testrbdnbd16809 2024-04-24T14:35:19.293 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:19.294 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:19.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:35:19.331 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:19.334+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:19.398 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:19.398 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:19.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:35:19.415 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-24T14:35:19.450 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:19.454+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:19.532 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:19.534+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:19.553 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:19.553 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:19.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:35:19.633 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:19.636+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:19.649 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:19.650 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:19.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:35:19.733 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:19.740+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:19.757 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:19.757 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:19.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:35:19.834 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:19.838+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:19.841 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.5 to 1 (10000) 2024-04-24T14:35:19.842 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2024-04-24T14:35:19.846 INFO:tasks.workunit.client.0.smithi173.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-04-24T14:35:19.856 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T14:35:19.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:35:19.859 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd -p rbd --namespace ns remove testrbdnbd16809 2024-04-24T14:35:19.867 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:19.867 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:19.877 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:35:19.934 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:19.938+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:20.027 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:20.027 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:20.035 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:20.040+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:20.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:35:20.051 INFO:tasks.workunit.client.0.smithi173.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete...2024-04-24T14:35:20.056+0000 7f128ed48700 0 -- 172.21.15.173:0/3999169704 >> [v2:172.21.15.103:6808/20318,v1:172.21.15.103:6809/20318] conn(0x7f1274008980 msgr2=0x7f127400ae00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-04-24T14:35:20.076 INFO:tasks.workunit.client.0.smithi173.stderr: Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-04-24T14:35:20.080+0000 7f128ed48700 0 -- 172.21.15.173:0/3999169704 >> [v2:172.21.15.103:6808/20318,v1:172.21.15.103:6809/20318] conn(0x7f1270038380 msgr2=0x7f127003a7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-04-24T14:35:20.091 INFO:tasks.workunit.client.0.smithi173.stderr: Removing image: 100% complete...done. 2024-04-24T14:35:20.098 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd namespace remove rbd/ns 2024-04-24T14:35:20.136 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:20.138+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:20.136 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:20.137 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:20.146 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:35:20.166 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:20.169 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:35:20.173 INFO:tasks.workunit.client.0.smithi173.stderr:+ rbd config global rm global rbd_default_pool 2024-04-24T14:35:20.181 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-04-24T14:35:19.838536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":226108825,"num_objects":121,"num_object_clones":18,"num_object_copies":242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":28,"num_read":7908,"num_read_kb":182303,"num_write":1024,"num_write_kb":273294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":445826026,"num_keys_recovered":150,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1094,"ondisk_log_size":1094,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":1943500,"kb_used_data":436240,"kb_used_omap":8,"kb_used_meta":1507191,"kb_avail":469915700,"statfs":{"total":483183820800,"available":481193676800,"internally_reserved":0,"allocated":446709760,"data_stored":453185629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8520,"internal_metadata":1543364280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.463012"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":121,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.360739+0000","last_change":"2024-04-24T14:35:04.316837+0000","last_active":"2024-04-24T14:35:14.360739+0000","last_peered":"2024-04-24T14:35:14.360739+0000","last_clean":"2024-04-24T14:35:14.360739+0000","last_became_active":"2024-04-24T14:35:04.316535+0000","last_became_peered":"2024-04-24T14:35:04.316535+0000","last_unstale":"2024-04-24T14:35:14.360739+0000","last_undegraded":"2024-04-24T14:35:14.360739+0000","last_fullsized":"2024-04-24T14:35:14.360739+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:51:46.418293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062826099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":258,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.360681+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:35:14.360681+0000","last_peered":"2024-04-24T14:35:14.360681+0000","last_clean":"2024-04-24T14:35:14.360681+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:35:14.360681+0000","last_undegraded":"2024-04-24T14:35:14.360681+0000","last_fullsized":"2024-04-24T14:35:14.360681+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"49'3","reported_seq":27,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.399219+0000","last_change":"2024-04-24T14:35:15.399469+0000","last_active":"2024-04-24T14:35:15.399219+0000","last_peered":"2024-04-24T14:35:15.399219+0000","last_clean":"2024-04-24T14:35:15.399219+0000","last_became_active":"2024-04-24T14:35:04.316770+0000","last_became_peered":"2024-04-24T14:35:04.316770+0000","last_unstale":"2024-04-24T14:35:15.399219+0000","last_undegraded":"2024-04-24T14:35:15.399219+0000","last_fullsized":"2024-04-24T14:35:15.399219+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:06.338365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049436100000000002,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.2","version":"114'163","reported_seq":532,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.407376+0000","last_change":"2024-04-24T14:35:15.405667+0000","last_active":"2024-04-24T14:35:15.407376+0000","last_peered":"2024-04-24T14:35:15.407376+0000","last_clean":"2024-04-24T14:35:15.407376+0000","last_became_active":"2024-04-24T14:35:04.319930+0000","last_became_peered":"2024-04-24T14:35:04.319930+0000","last_unstale":"2024-04-24T14:35:15.407376+0000","last_undegraded":"2024-04-24T14:35:15.407376+0000","last_fullsized":"2024-04-24T14:35:15.407376+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:52.537625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065911640000000001,"stat_sum":{"num_bytes":25165859,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":9,"num_read":1669,"num_read_kb":48605,"num_write":121,"num_write_kb":32866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":104988999,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.6","version":"74'1","reported_seq":98,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.361394+0000","last_change":"2024-04-24T14:34:42.212529+0000","last_active":"2024-04-24T14:35:14.361394+0000","last_peered":"2024-04-24T14:35:14.361394+0000","last_clean":"2024-04-24T14:35:14.361394+0000","last_became_active":"2024-04-24T14:34:42.205009+0000","last_became_peered":"2024-04-24T14:34:42.205009+0000","last_unstale":"2024-04-24T14:35:14.361394+0000","last_undegraded":"2024-04-24T14:35:14.361394+0000","last_fullsized":"2024-04-24T14:35:14.361394+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:53.534797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039681800000000001,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"108'4","reported_seq":160,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.375984+0000","last_change":"2024-04-24T14:35:15.376358+0000","last_active":"2024-04-24T14:35:15.375984+0000","last_peered":"2024-04-24T14:35:15.375984+0000","last_clean":"2024-04-24T14:35:15.375984+0000","last_became_active":"2024-04-24T14:34:42.206964+0000","last_became_peered":"2024-04-24T14:34:42.206964+0000","last_unstale":"2024-04-24T14:35:15.375984+0000","last_undegraded":"2024-04-24T14:35:15.375984+0000","last_fullsized":"2024-04-24T14:35:15.375984+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:59:11.424740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081610199999999997,"stat_sum":{"num_bytes":12582990,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":12289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.1","version":"114'403","reported_seq":2931,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.405413+0000","last_change":"2024-04-24T14:35:15.394187+0000","last_active":"2024-04-24T14:35:15.405413+0000","last_peered":"2024-04-24T14:35:15.405413+0000","last_clean":"2024-04-24T14:35:15.405413+0000","last_became_active":"2024-04-24T14:34:36.137865+0000","last_became_peered":"2024-04-24T14:34:36.137865+0000","last_unstale":"2024-04-24T14:35:15.405413+0000","last_undegraded":"2024-04-24T14:35:15.405413+0000","last_fullsized":"2024-04-24T14:35:15.405413+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:14.236340+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.013610193,"stat_sum":{"num_bytes":30433318,"num_objects":19,"num_object_clones":8,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":8,"num_read":4573,"num_read_kb":77587,"num_write":265,"num_write_kb":56438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":25165862,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.7","version":"0'0","reported_seq":15,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.557027+0000","last_change":"2024-04-24T14:35:04.316682+0000","last_active":"2024-04-24T14:35:14.557027+0000","last_peered":"2024-04-24T14:35:14.557027+0000","last_clean":"2024-04-24T14:35:14.557027+0000","last_became_active":"2024-04-24T14:35:04.315988+0000","last_became_peered":"2024-04-24T14:35:04.315988+0000","last_unstale":"2024-04-24T14:35:14.557027+0000","last_undegraded":"2024-04-24T14:35:14.557027+0000","last_fullsized":"2024-04-24T14:35:14.557027+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:51:37.583290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047250999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":171,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.375457+0000","last_change":"2024-04-24T14:35:15.375958+0000","last_active":"2024-04-24T14:35:15.375457+0000","last_peered":"2024-04-24T14:35:15.375457+0000","last_clean":"2024-04-24T14:35:15.375457+0000","last_became_active":"2024-04-24T14:34:42.207422+0000","last_became_peered":"2024-04-24T14:34:42.207422+0000","last_unstale":"2024-04-24T14:35:15.375457+0000","last_undegraded":"2024-04-24T14:35:15.375457+0000","last_fullsized":"2024-04-24T14:35:15.375457+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:02.081366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069472499999999999,"stat_sum":{"num_bytes":8388684,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":5,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.0","version":"114'260","reported_seq":740,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.383078+0000","last_change":"2024-04-24T14:35:15.383440+0000","last_active":"2024-04-24T14:35:15.383078+0000","last_peered":"2024-04-24T14:35:15.383078+0000","last_clean":"2024-04-24T14:35:15.383078+0000","last_became_active":"2024-04-24T14:34:44.841882+0000","last_became_peered":"2024-04-24T14:34:44.841882+0000","last_unstale":"2024-04-24T14:35:15.383078+0000","last_undegraded":"2024-04-24T14:35:15.383078+0000","last_fullsized":"2024-04-24T14:35:15.383078+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:47:40.933472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078879529999999996,"stat_sum":{"num_bytes":23117862,"num_objects":16,"num_object_clones":4,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":7,"num_read":995,"num_read_kb":38734,"num_write":337,"num_write_kb":34052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":43040844,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.4","version":"74'1","reported_seq":67,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.657283+0000","last_change":"2024-04-24T14:34:42.334431+0000","last_active":"2024-04-24T14:35:14.657283+0000","last_peered":"2024-04-24T14:35:14.657283+0000","last_clean":"2024-04-24T14:35:14.657283+0000","last_became_active":"2024-04-24T14:34:42.207515+0000","last_became_peered":"2024-04-24T14:34:42.207515+0000","last_unstale":"2024-04-24T14:35:14.657283+0000","last_undegraded":"2024-04-24T14:35:14.657283+0000","last_fullsized":"2024-04-24T14:35:14.657283+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:16:14.137756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032986300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"108'7","reported_seq":36,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.399122+0000","last_change":"2024-04-24T14:35:15.399521+0000","last_active":"2024-04-24T14:35:15.399122+0000","last_peered":"2024-04-24T14:35:15.399122+0000","last_clean":"2024-04-24T14:35:15.399122+0000","last_became_active":"2024-04-24T14:35:04.318211+0000","last_became_peered":"2024-04-24T14:35:04.318211+0000","last_unstale":"2024-04-24T14:35:15.399122+0000","last_undegraded":"2024-04-24T14:35:15.399122+0000","last_fullsized":"2024-04-24T14:35:15.399122+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:53:27.959804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056404399999999998,"stat_sum":{"num_bytes":20971598,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":13,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":83886080,"num_keys_recovered":14,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.3","version":"114'94","reported_seq":83,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.384757+0000","last_change":"2024-04-24T14:35:15.384757+0000","last_active":"2024-04-24T14:35:15.384757+0000","last_peered":"2024-04-24T14:35:15.384757+0000","last_clean":"2024-04-24T14:35:15.384757+0000","last_became_active":"2024-04-24T14:34:06.852598+0000","last_became_peered":"2024-04-24T14:34:06.852598+0000","last_unstale":"2024-04-24T14:35:15.384757+0000","last_undegraded":"2024-04-24T14:35:15.384757+0000","last_fullsized":"2024-04-24T14:35:15.384757+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:25:57.592466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044375819999999998,"stat_sum":{"num_bytes":12583042,"num_objects":12,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":4,"num_read":465,"num_read_kb":16976,"num_write":81,"num_write_kb":16421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777362,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.2","version":"53'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.556961+0000","last_change":"2024-04-24T14:35:04.394903+0000","last_active":"2024-04-24T14:35:14.556961+0000","last_peered":"2024-04-24T14:35:14.556961+0000","last_clean":"2024-04-24T14:35:14.556961+0000","last_became_active":"2024-04-24T14:35:04.318139+0000","last_became_peered":"2024-04-24T14:35:04.318139+0000","last_unstale":"2024-04-24T14:35:14.556961+0000","last_undegraded":"2024-04-24T14:35:14.556961+0000","last_fullsized":"2024-04-24T14:35:14.556961+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:37.590793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054416299999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"108'13","reported_seq":59,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.381744+0000","last_change":"2024-04-24T14:35:15.382258+0000","last_active":"2024-04-24T14:35:15.381744+0000","last_peered":"2024-04-24T14:35:15.381744+0000","last_clean":"2024-04-24T14:35:15.381744+0000","last_became_active":"2024-04-24T14:34:42.208181+0000","last_became_peered":"2024-04-24T14:34:42.208181+0000","last_unstale":"2024-04-24T14:35:15.381744+0000","last_undegraded":"2024-04-24T14:35:15.381744+0000","last_fullsized":"2024-04-24T14:35:15.381744+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:32:37.282127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020955409999999998,"stat_sum":{"num_bytes":37748852,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":16,"num_read_kb":135,"num_write":13,"num_write_kb":36868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":37748774,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.556878+0000","last_change":"2024-04-24T14:35:04.317455+0000","last_active":"2024-04-24T14:35:14.556878+0000","last_peered":"2024-04-24T14:35:14.556878+0000","last_clean":"2024-04-24T14:35:14.556878+0000","last_became_active":"2024-04-24T14:35:04.317129+0000","last_became_peered":"2024-04-24T14:35:04.317129+0000","last_unstale":"2024-04-24T14:35:14.556878+0000","last_undegraded":"2024-04-24T14:35:14.556878+0000","last_fullsized":"2024-04-24T14:35:14.556878+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:24:39.250920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030291500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"108'6","reported_seq":133,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.376658+0000","last_change":"2024-04-24T14:35:15.377184+0000","last_active":"2024-04-24T14:35:15.376658+0000","last_peered":"2024-04-24T14:35:15.376658+0000","last_clean":"2024-04-24T14:35:15.376658+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:35:15.376658+0000","last_undegraded":"2024-04-24T14:35:15.376658+0000","last_fullsized":"2024-04-24T14:35:15.376658+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014116879999999999,"stat_sum":{"num_bytes":20971598,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":128,"num_write":6,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.0","version":"74'1","reported_seq":84,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.563216+0000","last_change":"2024-04-24T14:34:36.137425+0000","last_active":"2024-04-24T14:35:14.563216+0000","last_peered":"2024-04-24T14:35:14.563216+0000","last_clean":"2024-04-24T14:35:14.563216+0000","last_became_active":"2024-04-24T14:34:36.136250+0000","last_became_peered":"2024-04-24T14:34:36.136250+0000","last_unstale":"2024-04-24T14:35:14.563216+0000","last_undegraded":"2024-04-24T14:35:14.563216+0000","last_fullsized":"2024-04-24T14:35:14.563216+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:32:13.633169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013794250000000001,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"76'5","reported_seq":104,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.376511+0000","last_change":"2024-04-24T14:35:15.376785+0000","last_active":"2024-04-24T14:35:15.376511+0000","last_peered":"2024-04-24T14:35:15.376511+0000","last_clean":"2024-04-24T14:35:15.376511+0000","last_became_active":"2024-04-24T14:34:42.208652+0000","last_became_peered":"2024-04-24T14:34:42.208652+0000","last_unstale":"2024-04-24T14:35:15.376511+0000","last_undegraded":"2024-04-24T14:35:15.376511+0000","last_fullsized":"2024-04-24T14:35:15.376511+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:13:41.703506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202633,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165862,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.1","version":"74'3","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.563265+0000","last_change":"2024-04-24T14:35:04.323260+0000","last_active":"2024-04-24T14:35:14.563265+0000","last_peered":"2024-04-24T14:35:14.563265+0000","last_clean":"2024-04-24T14:35:14.563265+0000","last_became_active":"2024-04-24T14:35:04.319983+0000","last_became_peered":"2024-04-24T14:35:04.319983+0000","last_unstale":"2024-04-24T14:35:14.563265+0000","last_undegraded":"2024-04-24T14:35:14.563265+0000","last_fullsized":"2024-04-24T14:35:14.563265+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:08.239076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000856414,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"84'6","reported_seq":168,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.375740+0000","last_change":"2024-04-24T14:35:15.376440+0000","last_active":"2024-04-24T14:35:15.375740+0000","last_peered":"2024-04-24T14:35:15.375740+0000","last_clean":"2024-04-24T14:35:15.375740+0000","last_became_active":"2024-04-24T14:34:06.852864+0000","last_became_peered":"2024-04-24T14:34:06.852864+0000","last_unstale":"2024-04-24T14:35:15.375740+0000","last_undegraded":"2024-04-24T14:35:15.375740+0000","last_fullsized":"2024-04-24T14:35:15.375740+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:42.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089422000000000004,"stat_sum":{"num_bytes":8388722,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":6,"num_write_kb":8196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":111,"num_keys_recovered":26,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":91300081,"num_objects":65,"num_object_clones":18,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":28,"num_read":7702,"num_read_kb":181902,"num_write":804,"num_write_kb":139777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":189973067,"num_keys_recovered":73,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192294912,"data_stored":202768904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":920,"ondisk_log_size":920,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":134218306,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":52,"num_read_kb":271,"num_write":55,"num_write_kb":131095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":255852848,"num_keys_recovered":51,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":373399552,"data_stored":373294364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49,"ondisk_log_size":49,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640296,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356100,"kb_used_data":54648,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94015740,"statfs":{"total":96636764160,"available":96272117760,"internally_reserved":0,"allocated":55959552,"data_stored":55166562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.80200000000000005,"15min":0.81499999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0720000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.80300000000000005,"15min":0.80900000000000005},"min":{"1min":0.42699999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.012,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.1679999999999999}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":1.2430000000000001}]},{"osd":2,"last update":"Wed Apr 24 14:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.82499999999999996,"15min":0.84099999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.95799999999999996,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.84199999999999997,"15min":0.85699999999999998},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.98599999999999999,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.0860000000000001}]},{"osd":3,"last update":"Wed Apr 24 14:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.80500000000000005,"15min":0.82499999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.93899999999999995,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.84499999999999997,"15min":0.86699999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0349999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.76800000000000002}]},{"osd":4,"last update":"Wed Apr 24 14:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.872,"15min":0.876},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.5030000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.96399999999999997,"15min":0.98799999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.083,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":1.022}]}]},{"osd":4,"up_from":11,"seq":47244640296,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422712,"kb_used_data":121260,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93949128,"statfs":{"total":96636764160,"available":96203907072,"internally_reserved":0,"allocated":124170240,"data_stored":128608238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.78400000000000003,"15min":0.78100000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.91000000000000003,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.80900000000000005}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.871}]},{"osd":2,"last update":"Wed Apr 24 14:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.84699999999999998,"15min":0.86499999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.95199999999999996,"5min":1.113,"15min":1.113},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.84699999999999998,"15min":0.85499999999999998},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.161,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Apr 24 14:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.84299999999999997,"15min":0.85499999999999998},"min":{"1min":0.56899999999999995,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.121,"5min":1.931,"15min":1.931},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.85999999999999999,"15min":0.874},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.393,"5min":1.591,"15min":1.591},"last":0.61499999999999999}]},{"osd":5,"last update":"Wed Apr 24 14:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.77400000000000002,"15min":0.78000000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.069,"5min":1.087,"15min":1.087},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.91100000000000003,"15min":0.92800000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1890000000000001,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.752}]}]},{"osd":3,"up_from":11,"seq":47244640294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377168,"kb_used_data":75716,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93994672,"statfs":{"total":96636764160,"available":96250544128,"internally_reserved":0,"allocated":77533184,"data_stored":76722474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.75900000000000001,"15min":0.77500000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.88500000000000001,"5min":1.282,"15min":1.282},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80200000000000005,"15min":0.80100000000000005},"min":{"1min":0.59299999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.131,"5min":1.208,"15min":1.208},"last":0.60799999999999998}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.64600000000000002}]},{"osd":2,"last update":"Wed Apr 24 14:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.86099999999999999,"15min":0.875},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.087,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.88,"5min":0.93200000000000005,"15min":0.93999999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.268,"5min":1.337,"15min":1.337},"last":0.76300000000000001}]},{"osd":4,"last update":"Wed Apr 24 14:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.71699999999999997,"15min":0.72199999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.93100000000000005,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.78500000000000003,"15min":0.79400000000000004},"min":{"1min":0.58499999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.92800000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.80400000000000005}]},{"osd":5,"last update":"Wed Apr 24 14:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78800000000000003,"15min":0.78600000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.022,"5min":1.367,"15min":1.367},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.86199999999999999,"15min":0.86899999999999999},"min":{"1min":0.52800000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.169,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.92100000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640296,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344820,"kb_used_data":43368,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94027020,"statfs":{"total":96636764160,"available":96283668480,"internally_reserved":0,"allocated":44408832,"data_stored":43615840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.499},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.59799999999999998,"15min":0.60899999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.873,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.52900000000000003}]},{"osd":1,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.56299999999999994}]},{"osd":3,"last update":"Wed Apr 24 14:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.84199999999999997,"15min":0.85399999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.014,"5min":1.147,"15min":1.147},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.89200000000000002,"15min":0.91100000000000003},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.111,"5min":1.77,"15min":1.77},"last":0.81599999999999995}]},{"osd":4,"last update":"Wed Apr 24 14:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.84699999999999998,"15min":0.85399999999999998},"min":{"1min":0.61299999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.0620000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.874,"15min":0.89200000000000002},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.995,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.76500000000000001}]},{"osd":5,"last update":"Wed Apr 24 14:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.93700000000000006,"15min":0.96099999999999997},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.113,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.90000000000000002,"15min":0.92000000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.097,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.93600000000000005}]}]},{"osd":1,"up_from":0,"seq":283467841552,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":11,"seq":47244640295,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":442700,"kb_used_data":141248,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93929140,"statfs":{"total":96636764160,"available":96183439360,"internally_reserved":0,"allocated":144637952,"data_stored":149072515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 14:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.60299999999999998}]},{"osd":2,"last update":"Wed Apr 24 14:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58899999999999997,"15min":0.59499999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67600000000000005,"15min":0.67800000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Apr 24 14:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79800000000000004,"15min":0.80000000000000004},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80200000000000005,"15min":0.80400000000000005},"min":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.496,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.78500000000000003}]},{"osd":4,"last update":"Wed Apr 24 14:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.91400000000000003,"15min":0.92400000000000004},"min":{"1min":0.62,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.91500000000000004,"15min":0.91800000000000004},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3919999999999999,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.71499999999999997}]},{"osd":5,"last update":"Wed Apr 24 14:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.88,"15min":0.88700000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.92100000000000004,"15min":0.92700000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.875}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100691968,"data_stored":100663722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104873984,"data_stored":104857752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20979712,"data_stored":20971596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":58744832,"data_stored":58720524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33558528,"data_stored":33554510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41992192,"data_stored":47243432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22073344,"data_stored":22069286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":22073344,"data_stored":22069286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":21045248,"data_stored":21037091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":64065536,"data_stored":69312718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":21045248,"data_stored":21037091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":51,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:35:20.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:35:20.217 INFO:tasks.workunit.client.0.smithi173.stderr:+ ceph osd pool delete new_default_pool16809 new_default_pool16809 --yes-i-really-really-mean-it 2024-04-24T14:35:20.237 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:20.240+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:20.254 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:20.254 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:20.263 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:35:20.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-24T14:35:20.367 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:20.368 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:20.374 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:20.378+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:20.376 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:35:20.439 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:20.442+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:20.456 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2024-04-24T14:35:20.467 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:20.469 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:35:20.476 INFO:teuthology.orchestra.run.smithi103.stdout:ERROR: (22) Invalid argument 2024-04-24T14:35:20.477 INFO:teuthology.orchestra.run.smithi103.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-24T14:35:20.478 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2024-04-24T14:35:20.480 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-04-24T14:35:19.838536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":226108825,"num_objects":121,"num_object_clones":18,"num_object_copies":242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":28,"num_read":7908,"num_read_kb":182303,"num_write":1024,"num_write_kb":273294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":445826026,"num_keys_recovered":150,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1094,"ondisk_log_size":1094,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":1943500,"kb_used_data":436240,"kb_used_omap":8,"kb_used_meta":1507191,"kb_avail":469915700,"statfs":{"total":483183820800,"available":481193676800,"internally_reserved":0,"allocated":446709760,"data_stored":453185629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8520,"internal_metadata":1543364280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.463012"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":121,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.360739+0000","last_change":"2024-04-24T14:35:04.316837+0000","last_active":"2024-04-24T14:35:14.360739+0000","last_peered":"2024-04-24T14:35:14.360739+0000","last_clean":"2024-04-24T14:35:14.360739+0000","last_became_active":"2024-04-24T14:35:04.316535+0000","last_became_peered":"2024-04-24T14:35:04.316535+0000","last_unstale":"2024-04-24T14:35:14.360739+0000","last_undegraded":"2024-04-24T14:35:14.360739+0000","last_fullsized":"2024-04-24T14:35:14.360739+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:51:46.418293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062826099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":258,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.360681+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:35:14.360681+0000","last_peered":"2024-04-24T14:35:14.360681+0000","last_clean":"2024-04-24T14:35:14.360681+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:35:14.360681+0000","last_undegraded":"2024-04-24T14:35:14.360681+0000","last_fullsized":"2024-04-24T14:35:14.360681+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"49'3","reported_seq":27,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.399219+0000","last_change":"2024-04-24T14:35:15.399469+0000","last_active":"2024-04-24T14:35:15.399219+0000","last_peered":"2024-04-24T14:35:15.399219+0000","last_clean":"2024-04-24T14:35:15.399219+0000","last_became_active":"2024-04-24T14:35:04.316770+0000","last_became_peered":"2024-04-24T14:35:04.316770+0000","last_unstale":"2024-04-24T14:35:15.399219+0000","last_undegraded":"2024-04-24T14:35:15.399219+0000","last_fullsized":"2024-04-24T14:35:15.399219+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:06.338365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049436100000000002,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.2","version":"114'163","reported_seq":532,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.407376+0000","last_change":"2024-04-24T14:35:15.405667+0000","last_active":"2024-04-24T14:35:15.407376+0000","last_peered":"2024-04-24T14:35:15.407376+0000","last_clean":"2024-04-24T14:35:15.407376+0000","last_became_active":"2024-04-24T14:35:04.319930+0000","last_became_peered":"2024-04-24T14:35:04.319930+0000","last_unstale":"2024-04-24T14:35:15.407376+0000","last_undegraded":"2024-04-24T14:35:15.407376+0000","last_fullsized":"2024-04-24T14:35:15.407376+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:52.537625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065911640000000001,"stat_sum":{"num_bytes":25165859,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":9,"num_read":1669,"num_read_kb":48605,"num_write":121,"num_write_kb":32866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":104988999,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.6","version":"74'1","reported_seq":98,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.361394+0000","last_change":"2024-04-24T14:34:42.212529+0000","last_active":"2024-04-24T14:35:14.361394+0000","last_peered":"2024-04-24T14:35:14.361394+0000","last_clean":"2024-04-24T14:35:14.361394+0000","last_became_active":"2024-04-24T14:34:42.205009+0000","last_became_peered":"2024-04-24T14:34:42.205009+0000","last_unstale":"2024-04-24T14:35:14.361394+0000","last_undegraded":"2024-04-24T14:35:14.361394+0000","last_fullsized":"2024-04-24T14:35:14.361394+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:53.534797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039681800000000001,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"108'4","reported_seq":160,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.375984+0000","last_change":"2024-04-24T14:35:15.376358+0000","last_active":"2024-04-24T14:35:15.375984+0000","last_peered":"2024-04-24T14:35:15.375984+0000","last_clean":"2024-04-24T14:35:15.375984+0000","last_became_active":"2024-04-24T14:34:42.206964+0000","last_became_peered":"2024-04-24T14:34:42.206964+0000","last_unstale":"2024-04-24T14:35:15.375984+0000","last_undegraded":"2024-04-24T14:35:15.375984+0000","last_fullsized":"2024-04-24T14:35:15.375984+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:59:11.424740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081610199999999997,"stat_sum":{"num_bytes":12582990,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":12289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.1","version":"114'403","reported_seq":2931,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.405413+0000","last_change":"2024-04-24T14:35:15.394187+0000","last_active":"2024-04-24T14:35:15.405413+0000","last_peered":"2024-04-24T14:35:15.405413+0000","last_clean":"2024-04-24T14:35:15.405413+0000","last_became_active":"2024-04-24T14:34:36.137865+0000","last_became_peered":"2024-04-24T14:34:36.137865+0000","last_unstale":"2024-04-24T14:35:15.405413+0000","last_undegraded":"2024-04-24T14:35:15.405413+0000","last_fullsized":"2024-04-24T14:35:15.405413+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:14.236340+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.013610193,"stat_sum":{"num_bytes":30433318,"num_objects":19,"num_object_clones":8,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":8,"num_read":4573,"num_read_kb":77587,"num_write":265,"num_write_kb":56438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":25165862,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.7","version":"0'0","reported_seq":15,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.557027+0000","last_change":"2024-04-24T14:35:04.316682+0000","last_active":"2024-04-24T14:35:14.557027+0000","last_peered":"2024-04-24T14:35:14.557027+0000","last_clean":"2024-04-24T14:35:14.557027+0000","last_became_active":"2024-04-24T14:35:04.315988+0000","last_became_peered":"2024-04-24T14:35:04.315988+0000","last_unstale":"2024-04-24T14:35:14.557027+0000","last_undegraded":"2024-04-24T14:35:14.557027+0000","last_fullsized":"2024-04-24T14:35:14.557027+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:51:37.583290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047250999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":171,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.375457+0000","last_change":"2024-04-24T14:35:15.375958+0000","last_active":"2024-04-24T14:35:15.375457+0000","last_peered":"2024-04-24T14:35:15.375457+0000","last_clean":"2024-04-24T14:35:15.375457+0000","last_became_active":"2024-04-24T14:34:42.207422+0000","last_became_peered":"2024-04-24T14:34:42.207422+0000","last_unstale":"2024-04-24T14:35:15.375457+0000","last_undegraded":"2024-04-24T14:35:15.375457+0000","last_fullsized":"2024-04-24T14:35:15.375457+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:02.081366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069472499999999999,"stat_sum":{"num_bytes":8388684,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":5,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.0","version":"114'260","reported_seq":740,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.383078+0000","last_change":"2024-04-24T14:35:15.383440+0000","last_active":"2024-04-24T14:35:15.383078+0000","last_peered":"2024-04-24T14:35:15.383078+0000","last_clean":"2024-04-24T14:35:15.383078+0000","last_became_active":"2024-04-24T14:34:44.841882+0000","last_became_peered":"2024-04-24T14:34:44.841882+0000","last_unstale":"2024-04-24T14:35:15.383078+0000","last_undegraded":"2024-04-24T14:35:15.383078+0000","last_fullsized":"2024-04-24T14:35:15.383078+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:47:40.933472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078879529999999996,"stat_sum":{"num_bytes":23117862,"num_objects":16,"num_object_clones":4,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":7,"num_read":995,"num_read_kb":38734,"num_write":337,"num_write_kb":34052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":43040844,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.4","version":"74'1","reported_seq":67,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.657283+0000","last_change":"2024-04-24T14:34:42.334431+0000","last_active":"2024-04-24T14:35:14.657283+0000","last_peered":"2024-04-24T14:35:14.657283+0000","last_clean":"2024-04-24T14:35:14.657283+0000","last_became_active":"2024-04-24T14:34:42.207515+0000","last_became_peered":"2024-04-24T14:34:42.207515+0000","last_unstale":"2024-04-24T14:35:14.657283+0000","last_undegraded":"2024-04-24T14:35:14.657283+0000","last_fullsized":"2024-04-24T14:35:14.657283+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:16:14.137756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032986300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"108'7","reported_seq":36,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.399122+0000","last_change":"2024-04-24T14:35:15.399521+0000","last_active":"2024-04-24T14:35:15.399122+0000","last_peered":"2024-04-24T14:35:15.399122+0000","last_clean":"2024-04-24T14:35:15.399122+0000","last_became_active":"2024-04-24T14:35:04.318211+0000","last_became_peered":"2024-04-24T14:35:04.318211+0000","last_unstale":"2024-04-24T14:35:15.399122+0000","last_undegraded":"2024-04-24T14:35:15.399122+0000","last_fullsized":"2024-04-24T14:35:15.399122+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:53:27.959804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056404399999999998,"stat_sum":{"num_bytes":20971598,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":13,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":83886080,"num_keys_recovered":14,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.3","version":"114'94","reported_seq":83,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.384757+0000","last_change":"2024-04-24T14:35:15.384757+0000","last_active":"2024-04-24T14:35:15.384757+0000","last_peered":"2024-04-24T14:35:15.384757+0000","last_clean":"2024-04-24T14:35:15.384757+0000","last_became_active":"2024-04-24T14:34:06.852598+0000","last_became_peered":"2024-04-24T14:34:06.852598+0000","last_unstale":"2024-04-24T14:35:15.384757+0000","last_undegraded":"2024-04-24T14:35:15.384757+0000","last_fullsized":"2024-04-24T14:35:15.384757+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:25:57.592466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044375819999999998,"stat_sum":{"num_bytes":12583042,"num_objects":12,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":4,"num_read":465,"num_read_kb":16976,"num_write":81,"num_write_kb":16421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777362,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.2","version":"53'2","reported_seq":19,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.556961+0000","last_change":"2024-04-24T14:35:04.394903+0000","last_active":"2024-04-24T14:35:14.556961+0000","last_peered":"2024-04-24T14:35:14.556961+0000","last_clean":"2024-04-24T14:35:14.556961+0000","last_became_active":"2024-04-24T14:35:04.318139+0000","last_became_peered":"2024-04-24T14:35:04.318139+0000","last_unstale":"2024-04-24T14:35:14.556961+0000","last_undegraded":"2024-04-24T14:35:14.556961+0000","last_fullsized":"2024-04-24T14:35:14.556961+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:28:37.590793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054416299999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"108'13","reported_seq":59,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.381744+0000","last_change":"2024-04-24T14:35:15.382258+0000","last_active":"2024-04-24T14:35:15.381744+0000","last_peered":"2024-04-24T14:35:15.381744+0000","last_clean":"2024-04-24T14:35:15.381744+0000","last_became_active":"2024-04-24T14:34:42.208181+0000","last_became_peered":"2024-04-24T14:34:42.208181+0000","last_unstale":"2024-04-24T14:35:15.381744+0000","last_undegraded":"2024-04-24T14:35:15.381744+0000","last_fullsized":"2024-04-24T14:35:15.381744+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:32:37.282127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020955409999999998,"stat_sum":{"num_bytes":37748852,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":16,"num_read_kb":135,"num_write":13,"num_write_kb":36868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":37748774,"num_keys_recovered":3,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.556878+0000","last_change":"2024-04-24T14:35:04.317455+0000","last_active":"2024-04-24T14:35:14.556878+0000","last_peered":"2024-04-24T14:35:14.556878+0000","last_clean":"2024-04-24T14:35:14.556878+0000","last_became_active":"2024-04-24T14:35:04.317129+0000","last_became_peered":"2024-04-24T14:35:04.317129+0000","last_unstale":"2024-04-24T14:35:14.556878+0000","last_undegraded":"2024-04-24T14:35:14.556878+0000","last_fullsized":"2024-04-24T14:35:14.556878+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:24:39.250920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030291500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"108'6","reported_seq":133,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.376658+0000","last_change":"2024-04-24T14:35:15.377184+0000","last_active":"2024-04-24T14:35:15.376658+0000","last_peered":"2024-04-24T14:35:15.376658+0000","last_clean":"2024-04-24T14:35:15.376658+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:35:15.376658+0000","last_undegraded":"2024-04-24T14:35:15.376658+0000","last_fullsized":"2024-04-24T14:35:15.376658+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014116879999999999,"stat_sum":{"num_bytes":20971598,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":128,"num_write":6,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.0","version":"74'1","reported_seq":84,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.563216+0000","last_change":"2024-04-24T14:34:36.137425+0000","last_active":"2024-04-24T14:35:14.563216+0000","last_peered":"2024-04-24T14:35:14.563216+0000","last_clean":"2024-04-24T14:35:14.563216+0000","last_became_active":"2024-04-24T14:34:36.136250+0000","last_became_peered":"2024-04-24T14:34:36.136250+0000","last_unstale":"2024-04-24T14:35:14.563216+0000","last_undegraded":"2024-04-24T14:35:14.563216+0000","last_fullsized":"2024-04-24T14:35:14.563216+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:32:13.633169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013794250000000001,"stat_sum":{"num_bytes":35,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"76'5","reported_seq":104,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.376511+0000","last_change":"2024-04-24T14:35:15.376785+0000","last_active":"2024-04-24T14:35:15.376511+0000","last_peered":"2024-04-24T14:35:15.376511+0000","last_clean":"2024-04-24T14:35:15.376511+0000","last_became_active":"2024-04-24T14:34:42.208652+0000","last_became_peered":"2024-04-24T14:34:42.208652+0000","last_unstale":"2024-04-24T14:35:15.376511+0000","last_undegraded":"2024-04-24T14:35:15.376511+0000","last_fullsized":"2024-04-24T14:35:15.376511+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:13:41.703506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001202633,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165862,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.1","version":"74'3","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-24T14:35:14.563265+0000","last_change":"2024-04-24T14:35:04.323260+0000","last_active":"2024-04-24T14:35:14.563265+0000","last_peered":"2024-04-24T14:35:14.563265+0000","last_clean":"2024-04-24T14:35:14.563265+0000","last_became_active":"2024-04-24T14:35:04.319983+0000","last_became_peered":"2024-04-24T14:35:04.319983+0000","last_unstale":"2024-04-24T14:35:14.563265+0000","last_undegraded":"2024-04-24T14:35:14.563265+0000","last_fullsized":"2024-04-24T14:35:14.563265+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:33:35.694794+0000","last_clean_scrub_stamp":"2024-04-24T14:33:35.694794+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:08.239076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000856414,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"84'6","reported_seq":168,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T14:35:15.375740+0000","last_change":"2024-04-24T14:35:15.376440+0000","last_active":"2024-04-24T14:35:15.375740+0000","last_peered":"2024-04-24T14:35:15.375740+0000","last_clean":"2024-04-24T14:35:15.375740+0000","last_became_active":"2024-04-24T14:34:06.852864+0000","last_became_peered":"2024-04-24T14:34:06.852864+0000","last_unstale":"2024-04-24T14:35:15.375740+0000","last_undegraded":"2024-04-24T14:35:15.375740+0000","last_fullsized":"2024-04-24T14:35:15.375740+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:42.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089422000000000004,"stat_sum":{"num_bytes":8388722,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":6,"num_write_kb":8196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":111,"num_keys_recovered":26,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":16,"acting":16,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":91300081,"num_objects":65,"num_object_clones":18,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":28,"num_read":7702,"num_read_kb":181902,"num_write":804,"num_write_kb":139777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":189973067,"num_keys_recovered":73,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192294912,"data_stored":202768904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":920,"ondisk_log_size":920,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":134218306,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":52,"num_read_kb":271,"num_write":55,"num_write_kb":131095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":255852848,"num_keys_recovered":51,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":373399552,"data_stored":373294364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49,"ondisk_log_size":49,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640296,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356100,"kb_used_data":54648,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94015740,"statfs":{"total":96636764160,"available":96272117760,"internally_reserved":0,"allocated":55959552,"data_stored":55166562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.80200000000000005,"15min":0.81499999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0720000000000001,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.80300000000000005,"15min":0.80900000000000005},"min":{"1min":0.42699999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.012,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.1679999999999999}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":1.2430000000000001}]},{"osd":2,"last update":"Wed Apr 24 14:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.82499999999999996,"15min":0.84099999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.95799999999999996,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.84199999999999997,"15min":0.85699999999999998},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.98599999999999999,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.0860000000000001}]},{"osd":3,"last update":"Wed Apr 24 14:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.80500000000000005,"15min":0.82499999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.93899999999999995,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.84499999999999997,"15min":0.86699999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0349999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.76800000000000002}]},{"osd":4,"last update":"Wed Apr 24 14:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.872,"15min":0.876},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.5030000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.96399999999999997,"15min":0.98799999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.083,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":1.022}]}]},{"osd":4,"up_from":11,"seq":47244640296,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422712,"kb_used_data":121260,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93949128,"statfs":{"total":96636764160,"available":96203907072,"internally_reserved":0,"allocated":124170240,"data_stored":128608238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.78400000000000003,"15min":0.78100000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.78800000000000003,"15min":0.80200000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.91000000000000003,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.80900000000000005}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.871}]},{"osd":2,"last update":"Wed Apr 24 14:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.84699999999999998,"15min":0.86499999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.95199999999999996,"5min":1.113,"15min":1.113},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.84699999999999998,"15min":0.85499999999999998},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.161,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Apr 24 14:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.84299999999999997,"15min":0.85499999999999998},"min":{"1min":0.56899999999999995,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.121,"5min":1.931,"15min":1.931},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.85999999999999999,"15min":0.874},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.393,"5min":1.591,"15min":1.591},"last":0.61499999999999999}]},{"osd":5,"last update":"Wed Apr 24 14:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.77400000000000002,"15min":0.78000000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.069,"5min":1.087,"15min":1.087},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.91100000000000003,"15min":0.92800000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.1890000000000001,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.752}]}]},{"osd":3,"up_from":11,"seq":47244640294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":377168,"kb_used_data":75716,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93994672,"statfs":{"total":96636764160,"available":96250544128,"internally_reserved":0,"allocated":77533184,"data_stored":76722474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.75900000000000001,"15min":0.77500000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.88500000000000001,"5min":1.282,"15min":1.282},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80200000000000005,"15min":0.80100000000000005},"min":{"1min":0.59299999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.131,"5min":1.208,"15min":1.208},"last":0.60799999999999998}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.64600000000000002}]},{"osd":2,"last update":"Wed Apr 24 14:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.86099999999999999,"15min":0.875},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.087,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.88,"5min":0.93200000000000005,"15min":0.93999999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.268,"5min":1.337,"15min":1.337},"last":0.76300000000000001}]},{"osd":4,"last update":"Wed Apr 24 14:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.71699999999999997,"15min":0.72199999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.93100000000000005,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.78500000000000003,"15min":0.79400000000000004},"min":{"1min":0.58499999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.92800000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.80400000000000005}]},{"osd":5,"last update":"Wed Apr 24 14:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78800000000000003,"15min":0.78600000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.022,"5min":1.367,"15min":1.367},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.86199999999999999,"15min":0.86899999999999999},"min":{"1min":0.52800000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.169,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.92100000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640296,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344820,"kb_used_data":43368,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94027020,"statfs":{"total":96636764160,"available":96283668480,"internally_reserved":0,"allocated":44408832,"data_stored":43615840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.499},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.59799999999999998,"15min":0.60899999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.873,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.52900000000000003}]},{"osd":1,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.56299999999999994}]},{"osd":3,"last update":"Wed Apr 24 14:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.84199999999999997,"15min":0.85399999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.014,"5min":1.147,"15min":1.147},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.89200000000000002,"15min":0.91100000000000003},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.111,"5min":1.77,"15min":1.77},"last":0.81599999999999995}]},{"osd":4,"last update":"Wed Apr 24 14:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.84699999999999998,"15min":0.85399999999999998},"min":{"1min":0.61299999999999999,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.0620000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.874,"15min":0.89200000000000002},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.995,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.76500000000000001}]},{"osd":5,"last update":"Wed Apr 24 14:34:24 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.93700000000000006,"15min":0.96099999999999997},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.113,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.90000000000000002,"15min":0.92000000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.097,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.93600000000000005}]}]},{"osd":1,"up_from":0,"seq":283467841552,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":11,"seq":47244640295,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":442700,"kb_used_data":141248,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":93929140,"statfs":{"total":96636764160,"available":96183439360,"internally_reserved":0,"allocated":144637952,"data_stored":149072515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 14:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.60299999999999998}]},{"osd":2,"last update":"Wed Apr 24 14:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58899999999999997,"15min":0.59499999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67600000000000005,"15min":0.67800000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Apr 24 14:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79800000000000004,"15min":0.80000000000000004},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80200000000000005,"15min":0.80400000000000005},"min":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.496,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.78500000000000003}]},{"osd":4,"last update":"Wed Apr 24 14:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.91400000000000003,"15min":0.92400000000000004},"min":{"1min":0.62,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.91500000000000004,"15min":0.91800000000000004},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3919999999999999,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.71499999999999997}]},{"osd":5,"last update":"Wed Apr 24 14:34:25 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.88,"15min":0.88700000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.92100000000000004,"15min":0.92700000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.875}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100691968,"data_stored":100663722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104873984,"data_stored":104857752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20979712,"data_stored":20971596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54550528,"data_stored":54526260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":58744832,"data_stored":58720524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33558528,"data_stored":33554510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41992192,"data_stored":47243432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22073344,"data_stored":22069286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":22073344,"data_stored":22069286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":21045248,"data_stored":21037091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":64065536,"data_stored":69312718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":21045248,"data_stored":21037091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":51,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":35,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:35:20.482 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T14:35:20.482 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-24T14:35:20.482 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-24T14:35:20.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-24T14:35:20.486 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:35:20.499 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2024-04-24T14:35:20.522 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2024-04-24T14:35:20.540 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:20.544+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:20.546 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2024-04-24T14:35:20.566 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2024-04-24T14:35:20.568 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-24T14:35:20.568 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-24T14:35:20.568 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-24T14:35:20.568 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-24T14:35:20.568 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-24T14:35:20.569 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-24T14:35:20.584 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:20.584 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:20.584 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:35:20.584 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:20.593 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:35:20.641 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:20.642+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:20.735 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:35:20.735 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:35:20.735 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:20.741 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:20.744+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:20.744 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:35:20.751 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:20.751 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:20.751 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 119, 2024-04-24T14:35:20.751 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "6706935e-d09a-472b-b839-321319fa9e54", 2024-04-24T14:35:20.751 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2024-04-24T14:32:09.059925+0000", 2024-04-24T14:35:20.751 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2024-04-24T14:35:19.834606+0000", 2024-04-24T14:35:20.751 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2024-04-24T14:34:00.856741+0000", 2024-04-24T14:35:20.751 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2024-04-24T14:35:03.287210+0000", 2024-04-24T14:35:20.751 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-24T14:35:20.751 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5801984, 2024-04-24T14:35:20.751 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2024-04-24T14:35:20.751 INFO:teuthology.orchestra.run.smithi090.stdout: "noscrub", 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 5, 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 4, 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 6, 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "luminous", 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "quincy", 2024-04-24T14:35:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:21.233022+0000", 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:35:20.753 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:35:20.754 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:35:20.754 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2024-04-24T14:35:20.754 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2024-04-24T14:35:20.754 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2024-04-24T14:35:20.754 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2024-04-24T14:35:20.754 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2024-04-24T14:35:20.754 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:35:20.754 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:35:20.754 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:35:20.754 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:35:20.754 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:35:20.754 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:35:20.755 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:35:20.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.755 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "14", 2024-04-24T14:35:20.755 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-24T14:35:20.755 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T14:35:20.755 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T14:35:20.755 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:35:20.755 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:35:20.755 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2024-04-24T14:35:20.755 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2024-04-24T14:35:20.755 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:35:20.755 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:35:20.755 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:35:20.755 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:35:20.756 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:35:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:35:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:35:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:35:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:35:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:35:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:35:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:35:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:35:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:35:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:35:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:35:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2024-04-24T14:35:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2024-04-24T14:35:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2024-04-24T14:35:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:35:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2024-04-24T14:35:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-24T14:35:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2024-04-24T14:35:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:26.609964+0000", 2024-04-24T14:35:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-24T14:35:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T14:35:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:35:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:35:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:35:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:35:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:35:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:35:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:35:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:35:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:35:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:35:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2024-04-24T14:35:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2024-04-24T14:35:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2024-04-24T14:35:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2024-04-24T14:35:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "119", 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "21", 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "21", 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "21", 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:35:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 13, 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 119, 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [ 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: 3 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": 3, 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": 3, 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:35:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:35:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:35:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:35:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:35:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:35:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:35:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:35:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:35:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:35:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:35:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:35:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:35:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:35:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:35:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:35:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:35:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:35:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:35:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:35:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:35:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:35:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:35:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "cache", 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:32:33.987564+0000", 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8201, 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 4, 2024-04-24T14:35:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 4, 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 4, 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 4, 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 4, 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "119", 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "21", 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "21", 2024-04-24T14:35:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "21", 2024-04-24T14:35:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:35:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:35:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 13, 2024-04-24T14:35:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 119, 2024-04-24T14:35:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:35:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:35:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:35:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:35:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:35:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": 2, 2024-04-24T14:35:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:35:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:35:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "writeback", 2024-04-24T14:35:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:35:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 250, 2024-04-24T14:35:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:35:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:35:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:35:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:35:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:35:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:35:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:35:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "bloom", 2024-04-24T14:35:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "false_positive_probability": 0.050000000000000003, 2024-04-24T14:35:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "target_size": 0, 2024-04-24T14:35:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "seed": 0 2024-04-24T14:35:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 60, 2024-04-24T14:35:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 8, 2024-04-24T14:35:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:35:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:35:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:35:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:35:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:35:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [ 2024-04-24T14:35:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:35:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:35:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:35:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:35:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:35:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:35:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000, 2024-04-24T14:35:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: 1000000 2024-04-24T14:35:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:35:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:35:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:35:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:35:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:35:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:35:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 4, 2024-04-24T14:35:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "new_default_pool16809", 2024-04-24T14:35:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2024-04-24T14:33:35.603212+0000", 2024-04-24T14:35:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 8193, 2024-04-24T14:35:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-24T14:35:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2024-04-24T14:35:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 2, 2024-04-24T14:35:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 1, 2024-04-24T14:35:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2024-04-24T14:35:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2024-04-24T14:35:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2024-04-24T14:35:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2024-04-24T14:35:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-24T14:35:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2024-04-24T14:35:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2024-04-24T14:35:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2024-04-24T14:35:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2024-04-24T14:35:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2024-04-24T14:35:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2024-04-24T14:35:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2024-04-24T14:35:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2024-04-24T14:35:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2024-04-24T14:35:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "53", 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 3, 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 53, 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2024-04-24T14:35:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2024-04-24T14:35:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2024-04-24T14:35:20.774 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2024-04-24T14:35:20.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.774 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2024-04-24T14:35:20.774 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2024-04-24T14:35:20.774 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2024-04-24T14:35:20.774 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2024-04-24T14:35:20.774 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2024-04-24T14:35:20.774 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2024-04-24T14:35:20.774 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2024-04-24T14:35:20.774 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2024-04-24T14:35:20.774 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2024-04-24T14:35:20.774 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2024-04-24T14:35:20.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2024-04-24T14:35:20.775 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2024-04-24T14:35:20.775 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2024-04-24T14:35:20.775 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2024-04-24T14:35:20.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.775 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:20.775 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2024-04-24T14:35:20.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.775 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-24T14:35:20.775 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "a2506b4f-3486-4f51-972f-1d9e4b015346", 2024-04-24T14:35:20.775 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:35:20.775 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:35:20.775 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.484344482421875, 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 96, 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.777 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2024-04-24T14:35:20.778 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:20.778 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.778 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.778 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:35:20.778 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.778 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.778 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2024-04-24T14:35:20.778 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:20.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.778 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.778 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2024-04-24T14:35:20.779 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:20.779 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.779 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.779 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:35:20.779 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.780 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.780 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2024-04-24T14:35:20.780 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:20.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.780 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.780 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2024-04-24T14:35:20.781 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20309 2024-04-24T14:35:20.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.781 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.781 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6817/20309", 2024-04-24T14:35:20.781 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6819/20309", 2024-04-24T14:35:20.781 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/20309", 2024-04-24T14:35:20.781 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6821/20309", 2024-04-24T14:35:20.781 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:35:20.782 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:35:20.782 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:35:20.782 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.782 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-24T14:35:20.782 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "9e17324f-a615-494f-ba0d-7c8aa8dac95c", 2024-04-24T14:35:20.782 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:35:20.782 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2024-04-24T14:35:20.782 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2024-04-24T14:35:20.782 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.2843475341796875, 2024-04-24T14:35:20.782 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 11, 2024-04-24T14:35:20.783 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 31, 2024-04-24T14:35:20.783 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 66, 2024-04-24T14:35:20.783 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 93, 2024-04-24T14:35:20.783 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 32, 2024-04-24T14:35:20.783 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:35:20.783 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:35:20.783 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.783 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.783 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2024-04-24T14:35:20.783 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:20.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.784 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.784 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2024-04-24T14:35:20.784 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:20.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.784 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.784 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:35:20.784 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.785 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.785 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2024-04-24T14:35:20.785 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:20.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.785 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.785 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2024-04-24T14:35:20.785 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:20.785 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.785 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.785 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:35:20.785 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.786 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.786 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2024-04-24T14:35:20.786 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:20.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.786 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.786 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2024-04-24T14:35:20.786 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:20.786 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.786 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.786 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:35:20.787 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.787 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.787 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2024-04-24T14:35:20.787 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:20.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.787 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.787 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2024-04-24T14:35:20.787 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 34413 2024-04-24T14:35:20.788 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.788 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.788 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/34413", 2024-04-24T14:35:20.788 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/34413", 2024-04-24T14:35:20.789 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/34413", 2024-04-24T14:35:20.789 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/34413", 2024-04-24T14:35:20.789 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:35:20.789 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:35:20.789 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:35:20.789 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.790 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-24T14:35:20.790 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7", 2024-04-24T14:35:20.790 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:35:20.790 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:35:20.790 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:35:20.790 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.8896331787109375, 2024-04-24T14:35:20.790 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:35:20.790 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 99, 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:20.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.793 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:35:20.794 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.794 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.794 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2024-04-24T14:35:20.794 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:20.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.794 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.794 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2024-04-24T14:35:20.794 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20310 2024-04-24T14:35:20.794 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.794 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.795 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6809/20310", 2024-04-24T14:35:20.795 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6811/20310", 2024-04-24T14:35:20.795 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6815/20310", 2024-04-24T14:35:20.795 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6813/20310", 2024-04-24T14:35:20.795 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:35:20.795 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:35:20.795 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:35:20.795 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.795 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-24T14:35:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "509c16da-22b2-48e8-aa51-c320c8c4fd8b", 2024-04-24T14:35:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:35:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:35:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:35:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:35:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:35:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:35:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:35:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 107, 2024-04-24T14:35:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:35:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:35:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:35:20.796 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.797 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.797 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6800", 2024-04-24T14:35:20.797 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:20.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.797 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.797 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6801", 2024-04-24T14:35:20.797 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:20.797 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.797 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.797 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:35:20.797 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.798 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.798 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6802", 2024-04-24T14:35:20.798 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:20.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.798 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.798 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6803", 2024-04-24T14:35:20.798 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:20.798 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:35:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6806", 2024-04-24T14:35:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6807", 2024-04-24T14:35:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:20.799 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6804", 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6805", 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20316 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.800 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.801 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6801/20316", 2024-04-24T14:35:20.801 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6803/20316", 2024-04-24T14:35:20.801 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6807/20316", 2024-04-24T14:35:20.801 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6805/20316", 2024-04-24T14:35:20.801 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:35:20.801 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:35:20.801 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:35:20.801 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.802 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-24T14:35:20.802 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1462066b-4e59-46d4-9b85-1f61ee1d56a6", 2024-04-24T14:35:20.802 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:35:20.802 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:35:20.802 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:35:20.802 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:35:20.802 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:35:20.802 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:35:20.802 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:35:20.803 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 107, 2024-04-24T14:35:20.803 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:35:20.803 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:35:20.803 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:35:20.803 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.803 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.803 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6808", 2024-04-24T14:35:20.803 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:20.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.803 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6809", 2024-04-24T14:35:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:35:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6810", 2024-04-24T14:35:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6811", 2024-04-24T14:35:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:35:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6814", 2024-04-24T14:35:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6815", 2024-04-24T14:35:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:35:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6812", 2024-04-24T14:35:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6813", 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20318 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6809/20318", 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6811/20318", 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6815/20318", 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6813/20318", 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:35:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-24T14:35:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "4dc954a8-5f3c-4826-b9bc-98cf30519382", 2024-04-24T14:35:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2024-04-24T14:35:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2024-04-24T14:35:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2024-04-24T14:35:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2024-04-24T14:35:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2024-04-24T14:35:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2024-04-24T14:35:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 11, 2024-04-24T14:35:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 107, 2024-04-24T14:35:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2024-04-24T14:35:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2024-04-24T14:35:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2024-04-24T14:35:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6816", 2024-04-24T14:35:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6817", 2024-04-24T14:35:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2024-04-24T14:35:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6818", 2024-04-24T14:35:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6819", 2024-04-24T14:35:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2024-04-24T14:35:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6822", 2024-04-24T14:35:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6823", 2024-04-24T14:35:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2024-04-24T14:35:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2024-04-24T14:35:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2024-04-24T14:35:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6820", 2024-04-24T14:35:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-04-24T14:35:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.103:6821", 2024-04-24T14:35:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 20317 2024-04-24T14:35:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.103:6817/20317", 2024-04-24T14:35:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.103:6819/20317", 2024-04-24T14:35:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.103:6823/20317", 2024-04-24T14:35:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.103:6821/20317", 2024-04-24T14:35:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2024-04-24T14:35:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2024-04-24T14:35:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2024-04-24T14:35:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2024-04-24T14:35:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2024-04-24T14:35:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:35:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:35:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:35:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:35:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:35:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.792899+0000", 2024-04-24T14:35:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:35:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2024-04-24T14:35:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:35:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:35:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:35:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:35:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2024-04-24T14:35:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.580414+0000", 2024-04-24T14:35:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 32 2024-04-24T14:35:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2024-04-24T14:35:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:35:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:35:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:35:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:35:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:35:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.713336+0000", 2024-04-24T14:35:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:35:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2024-04-24T14:35:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:35:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:35:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.579638+0000", 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.570510+0000", 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:35:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2024-04-24T14:35:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2024-04-24T14:35:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2024-04-24T14:35:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2024-04-24T14:35:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138320759226367, 2024-04-24T14:35:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2024-04-24T14:35:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2024-04-24T14:32:16.736558+0000", 2024-04-24T14:35:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2024-04-24T14:35:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2024-04-24T14:35:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [ 2024-04-24T14:35:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "4.3", 2024-04-24T14:35:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2024-04-24T14:35:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 0, 2024-04-24T14:35:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 5 2024-04-24T14:35:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2024-04-24T14:35:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2024-04-24T14:35:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2024-04-24T14:35:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2024-04-24T14:35:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2024-04-24T14:35:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2024-04-24T14:35:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2024-04-24T14:35:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2024-04-24T14:35:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2024-04-24T14:35:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2024-04-24T14:35:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2024-04-24T14:35:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2024-04-24T14:35:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-24T14:35:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-24T14:35:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 5, 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 5, 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [ 2024-04-24T14:35:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2024-04-24T14:35:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-24T14:35:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 5, 2024-04-24T14:35:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-24T14:35:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2024-04-24T14:35:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2024-04-24T14:35:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 5, 2024-04-24T14:35:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2024-04-24T14:35:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2024-04-24T14:35:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2024-04-24T14:35:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2024-04-24T14:35:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2024-04-24T14:35:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2024-04-24T14:35:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2024-04-24T14:35:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2024-04-24T14:35:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2024-04-24T14:35:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2024-04-24T14:35:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2024-04-24T14:35:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:20.824 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:20.826 INFO:tasks.thrashosds.thrasher:j is {'epoch': 119, 'fsid': '6706935e-d09a-472b-b839-321319fa9e54', 'created': '2024-04-24T14:32:09.059925+0000', 'modified': '2024-04-24T14:35:19.834606+0000', 'last_up_change': '2024-04-24T14:34:00.856741+0000', 'last_in_change': '2024-04-24T14:35:03.287210+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T14:32:21.233022+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-24T14:32:26.609964+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '21', 'last_force_op_resend_prenautilus': '21', 'last_force_op_resend_preluminous': '21', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 13, 'snap_epoch': 119, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [3], 'tier_of': -1, 'read_tier': 3, 'write_tier': 3, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cache', 'create_time': '2024-04-24T14:32:33.987564+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '21', 'last_force_op_resend_prenautilus': '21', 'last_force_op_resend_preluminous': '21', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 13, 'snap_epoch': 119, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 2, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 250, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'bloom', 'false_positive_probability': 0.05, 'target_size': 0, 'seed': 0}, 'hit_set_period': 60, 'hit_set_count': 8, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000, 1000000], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'new_default_pool16809', 'create_time': '2024-04-24T14:33:35.603212+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 53, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'a2506b4f-3486-4f51-972f-1d9e4b015346', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.484344482421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 20309}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6819', 'nonce': 20309}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 20309}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6820', 'nonce': 20309}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 20309}]}, 'public_addr': '172.21.15.90:6817/20309', 'cluster_addr': '172.21.15.90:6819/20309', 'heartbeat_back_addr': '172.21.15.90:6823/20309', 'heartbeat_front_addr': '172.21.15.90:6821/20309', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '9e17324f-a615-494f-ba0d-7c8aa8dac95c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.2843475341796875, 'last_clean_begin': 11, 'last_clean_end': 31, 'up_from': 66, 'up_thru': 93, 'down_at': 32, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 34413}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 34413}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 34413}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 34413}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 34413}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 34413}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 34413}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 34413}]}, 'public_addr': '172.21.15.90:6801/34413', 'cluster_addr': '172.21.15.90:6803/34413', 'heartbeat_back_addr': '172.21.15.90:6807/34413', 'heartbeat_front_addr': '172.21.15.90:6805/34413', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8896331787109375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6809', 'nonce': 20310}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6810', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 20310}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6814', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 20310}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 20310}, {'type': 'v1', 'addr': '172.21.15.90:6813', 'nonce': 20310}]}, 'public_addr': '172.21.15.90:6809/20310', 'cluster_addr': '172.21.15.90:6811/20310', 'heartbeat_back_addr': '172.21.15.90:6815/20310', 'heartbeat_front_addr': '172.21.15.90:6813/20310', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '509c16da-22b2-48e8-aa51-c320c8c4fd8b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6800', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6801', 'nonce': 20316}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6802', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6803', 'nonce': 20316}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6806', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6807', 'nonce': 20316}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6804', 'nonce': 20316}, {'type': 'v1', 'addr': '172.21.15.103:6805', 'nonce': 20316}]}, 'public_addr': '172.21.15.103:6801/20316', 'cluster_addr': '172.21.15.103:6803/20316', 'heartbeat_back_addr': '172.21.15.103:6807/20316', 'heartbeat_front_addr': '172.21.15.103:6805/20316', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1462066b-4e59-46d4-9b85-1f61ee1d56a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6808', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6809', 'nonce': 20318}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6810', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6811', 'nonce': 20318}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6814', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6815', 'nonce': 20318}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6812', 'nonce': 20318}, {'type': 'v1', 'addr': '172.21.15.103:6813', 'nonce': 20318}]}, 'public_addr': '172.21.15.103:6809/20318', 'cluster_addr': '172.21.15.103:6811/20318', 'heartbeat_back_addr': '172.21.15.103:6815/20318', 'heartbeat_front_addr': '172.21.15.103:6813/20318', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4dc954a8-5f3c-4826-b9bc-98cf30519382', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 107, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6816', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6817', 'nonce': 20317}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6818', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6819', 'nonce': 20317}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6822', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6823', 'nonce': 20317}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6820', 'nonce': 20317}, {'type': 'v1', 'addr': '172.21.15.103:6821', 'nonce': 20317}]}, 'public_addr': '172.21.15.103:6817/20317', 'cluster_addr': '172.21.15.103:6819/20317', 'heartbeat_back_addr': '172.21.15.103:6823/20317', 'heartbeat_front_addr': '172.21.15.103:6821/20317', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.792899+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.580414+0000', 'dead_epoch': 32}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.713336+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.579638+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.570510+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-24T14:32:16.736558+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.3', 'mappings': [{'from': 0, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 5, 'length': 4}]}, {'pool': 3, 'snaps': [{'begin': 5, 'length': 4}]}], 'new_removed_snaps': [{'pool': 2, 'snaps': [{'begin': 5, 'length': 1}]}, {'pool': 3, 'snaps': [{'begin': 5, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-24T14:35:20.826 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-24T14:35:20.842 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:20.844+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:20.846 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:20.846 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:35:20.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:35:20.847 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:35:20.847 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.847 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:20.847 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:20.847 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.185726497000005, 2024-04-24T14:35:20.847 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:35:20.847 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.847 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:20.847 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:20.847 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:20.847 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:20.847 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:35:20.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:20.848 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:20.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:35:20.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:35:20.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.849 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.849 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:35:20.849 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:35:20.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.849 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:35:20.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:35:20.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:35:20.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:35:20.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.850 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.850 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.850 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:35:20.850 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:35:20.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.851 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.851 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:35:20.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:35:20.851 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.851 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.851 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.851 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.851 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:20.851 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:20.851 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.182667785000007, 2024-04-24T14:35:20.851 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:20.852 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:35:20.853 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:35:20.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:35:20.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:35:20.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:35:20.854 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:35:20.854 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.854 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.854 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.854 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:35:20.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:35:20.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.855 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.855 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:35:20.855 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:35:20.855 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.855 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.855 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.855 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.855 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:20.855 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.182650401000004, 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.856 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.857 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:35:20.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:35:20.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:35:20.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:35:20.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.858 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.858 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.858 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:35:20.858 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:35:20.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:35:20.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:35:20.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.859 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.859 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.859 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:35:20.859 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:35:20.859 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.859 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.859 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.168214309000007, 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.860 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.861 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:35:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:35:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:35:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:35:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:35:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:35:20.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:35:20.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:35:20.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:35:20.863 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:35:20.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.863 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.863 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.863 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:35:20.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:35:20.864 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.864 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.864 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.864 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.864 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:20.864 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:35:20.864 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.168182009999995, 2024-04-24T14:35:20.864 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:35:20.864 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.864 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:20.864 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:20.864 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:20.865 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:20.865 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:35:20.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.865 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:35:20.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.865 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.865 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:35:20.865 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:35:20.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:20.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:35:20.866 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:35:20.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.866 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.866 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.866 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.867 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.153900633000006, 2024-04-24T14:35:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.869 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:35:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:35:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.870 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:35:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:35:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:35:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:35:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:35:20.871 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:35:20.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:35:20.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:35:20.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.872 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.872 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.872 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:35:20.872 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:35:20.872 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.872 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.872 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.873 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:20.873 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:20.873 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.153874703, 2024-04-24T14:35:20.873 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:35:20.873 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.873 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:20.873 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:20.873 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:20.873 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:20.873 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:35:20.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.873 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.873 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:20.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:20.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.874 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.874 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.874 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:20.874 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:20.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:35:20.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:35:20.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:35:20.875 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:35:20.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.875 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.875 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.875 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:35:20.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:35:20.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:35:20.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:35:20.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.876 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:35:20.876 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:35:20.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.876 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.876 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.877 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:35:20.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:35:20.877 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.877 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.877 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.877 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:20.877 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:35:20.877 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.138122307000003, 2024-04-24T14:35:20.877 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:35:20.877 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.877 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:35:20.877 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:20.878 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:20.878 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:20.878 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:35:20.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.878 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:35:20.878 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.878 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.878 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.878 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.879 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:35:20.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:35:20.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.880 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:35:20.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:35:20.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.880 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.880 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.880 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:35:20.880 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.138054791000002, 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:35:20.881 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.882 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:35:20.882 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:20.882 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:20.882 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:20.882 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:35:20.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.882 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.882 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.882 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:35:20.882 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.882 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.883 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.884 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.885 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:35:20.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:35:20.885 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.885 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.885 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.885 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.885 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:35:20.885 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:20.885 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 99.080641908000004, 2024-04-24T14:35:20.885 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:35:20.885 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.885 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:20.886 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:20.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.886 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.886 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:20.886 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.886 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:20.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:20.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:35:20.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:35:20.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.887 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.887 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.887 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:35:20.887 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:35:20.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:35:20.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:35:20.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.888 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:35:20.888 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:35:20.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.888 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.888 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:35:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:35:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:35:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:35:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:35:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:35:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.889 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.890 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.890 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:103 3.1 e87/74 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head v 87'261, mlcod=87'258)", 2024-04-24T14:35:20.890 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:35:20.890 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 53.889360703999998, 2024-04-24T14:35:20.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019745728000000001, 2024-04-24T14:35:20.890 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.890 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:20.890 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.890 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.890 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.890 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:35:20.890 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960679+0000", 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960692+0000", 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3709999999999999e-05 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.891 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960722+0000", 2024-04-24T14:35:20.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9791999999999999e-05 2024-04-24T14:35:20.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960733+0000", 2024-04-24T14:35:20.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.128e-05 2024-04-24T14:35:20.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.892 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.892 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.892 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960857+0000", 2024-04-24T14:35:20.892 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000123981 2024-04-24T14:35:20.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960926+0000", 2024-04-24T14:35:20.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8881999999999995e-05 2024-04-24T14:35:20.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:20.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980353+0000", 2024-04-24T14:35:20.893 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019426853000000001 2024-04-24T14:35:20.893 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.893 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.893 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.893 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980426+0000", 2024-04-24T14:35:20.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2803000000000001e-05 2024-04-24T14:35:20.894 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.894 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.894 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.894 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:105 3.1 e87/74 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head v 87'263, mlcod=87'258)", 2024-04-24T14:35:20.894 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:35:20.894 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 53.880234170000001, 2024-04-24T14:35:20.894 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014864743999999999, 2024-04-24T14:35:20.894 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.894 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:20.894 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969805+0000", 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.895 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969820+0000", 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5116999999999999e-05 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969852+0000", 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2135999999999999e-05 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969864+0000", 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.189e-05 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.896 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972122+0000", 2024-04-24T14:35:20.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022580389999999999 2024-04-24T14:35:20.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972165+0000", 2024-04-24T14:35:20.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2429000000000003e-05 2024-04-24T14:35:20.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:20.897 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984619+0000", 2024-04-24T14:35:20.897 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012454735 2024-04-24T14:35:20.897 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.897 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.897 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.898 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984672+0000", 2024-04-24T14:35:20.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2077999999999998e-05 2024-04-24T14:35:20.898 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.898 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.898 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.898 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.898 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:112 3.1 e87/74 3:a983798d:::rbd_data.12ff64619e2d.0000000000000006:head v 87'265, mlcod=87'258)", 2024-04-24T14:35:20.898 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:35:20.898 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 53.869895333999999, 2024-04-24T14:35:20.898 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012821978, 2024-04-24T14:35:20.898 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.898 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:20.898 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980144+0000", 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:20.899 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980159+0000", 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4769999999999999e-05 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980189+0000", 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0419000000000001e-05 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980213+0000", 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4036000000000002e-05 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.900 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980452+0000", 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00023921500000000001 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980468+0000", 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5843999999999999e-05 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992932+0000", 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012463314 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992968+0000", 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6075999999999998e-05 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:113 3.1 e87/74 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head v 87'267, mlcod=87'258)", 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 53.866360610000001, 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010561022999999999, 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.902 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983679+0000", 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.903 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983699+0000", 2024-04-24T14:35:20.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0392000000000001e-05 2024-04-24T14:35:20.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983737+0000", 2024-04-24T14:35:20.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8340000000000002e-05 2024-04-24T14:35:20.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.904 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.904 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983756+0000", 2024-04-24T14:35:20.904 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8402000000000001e-05 2024-04-24T14:35:20.904 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.904 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984706+0000", 2024-04-24T14:35:20.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095024400000000005 2024-04-24T14:35:20.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984741+0000", 2024-04-24T14:35:20.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4721000000000002e-05 2024-04-24T14:35:20.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.905 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.905 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:20.905 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994211+0000", 2024-04-24T14:35:20.905 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094708559999999997 2024-04-24T14:35:20.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.906 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.906 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994241+0000", 2024-04-24T14:35:20.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9941e-05 2024-04-24T14:35:20.906 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.906 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.906 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.906 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.906 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:115 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:20.906 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:35:20.906 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 53.865906572999997, 2024-04-24T14:35:20.906 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010347242, 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.907 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:35:20.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984132+0000", 2024-04-24T14:35:20.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:20.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.908 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984142+0000", 2024-04-24T14:35:20.908 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7370000000000001e-06 2024-04-24T14:35:20.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.908 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.908 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984154+0000", 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2288e-05 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984163+0000", 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3749999999999992e-06 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984246+0000", 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2962999999999998e-05 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.909 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984473+0000", 2024-04-24T14:35:20.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00022696399999999999 2024-04-24T14:35:20.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:20.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984771+0000", 2024-04-24T14:35:20.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00029739499999999999 2024-04-24T14:35:20.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:20.910 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992998+0000", 2024-04-24T14:35:20.910 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082275750000000009 2024-04-24T14:35:20.910 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.910 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.910 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994299+0000", 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013005289999999999 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994403+0000", 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010376499999999999 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994482+0000", 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9013999999999999e-05 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.912 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.912 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:128 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:20.912 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:35:20.912 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 53.844715645000001, 2024-04-24T14:35:20.912 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011784144, 2024-04-24T14:35:20.912 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.912 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:20.912 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:20.912 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:35:20.912 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:20.912 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 128 2024-04-24T14:35:20.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.912 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005324+0000", 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.913 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005332+0000", 2024-04-24T14:35:20.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2989999999999997e-06 2024-04-24T14:35:20.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005354+0000", 2024-04-24T14:35:20.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1705999999999999e-05 2024-04-24T14:35:20.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.914 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.914 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.914 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005362+0000", 2024-04-24T14:35:20.914 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5599999999999996e-06 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005418+0000", 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6685999999999999e-05 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005535+0000", 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117008 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005732+0000", 2024-04-24T14:35:20.915 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019718700000000001 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016440+0000", 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010707787999999999 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016913+0000", 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047270400000000001 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017047+0000", 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013434699999999999 2024-04-24T14:35:20.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017109+0000", 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2192000000000004e-05 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:130 3.1 e87/74 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 87'269, mlcod=87'265)", 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 53.844262884999999, 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013945631999999999, 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.917 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:35:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:35:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.918 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005777+0000", 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005791+0000", 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3743999999999999e-05 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005803+0000", 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2330999999999999e-05 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.919 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005810+0000", 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8990000000000004e-06 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005888+0000", 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7872000000000003e-05 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005911+0000", 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3156e-05 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019642+0000", 2024-04-24T14:35:20.920 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013730456 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019724+0000", 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1981e-05 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.2.0:211 3.0 e116/99 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:7 v 116'270, mlcod=115'266)", 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:17.397582+0000", 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.4524593889999999, 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011810169000000001, 2024-04-24T14:35:20.921 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397582+0000", 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397582+0000", 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.922 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397581+0000", 2024-04-24T14:35:20.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:20.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397596+0000", 2024-04-24T14:35:20.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.485e-05 2024-04-24T14:35:20.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.923 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.923 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397625+0000", 2024-04-24T14:35:20.923 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9892000000000001e-05 2024-04-24T14:35:20.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.923 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397647+0000", 2024-04-24T14:35:20.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1733e-05 2024-04-24T14:35:20.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397773+0000", 2024-04-24T14:35:20.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000126277 2024-04-24T14:35:20.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.924 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.924 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.924 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397850+0000", 2024-04-24T14:35:20.924 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6339000000000002e-05 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.409320+0000", 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011470556 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.409392+0000", 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1471999999999996e-05 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.925 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.926 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5795.0:56 3.1 3:990e5750:::rbd_header.12ff64619e2d:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e116)", 2024-04-24T14:35:20.926 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:17.399479+0000", 2024-04-24T14:35:20.926 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.4505621619999998, 2024-04-24T14:35:20.926 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013769491, 2024-04-24T14:35:20.926 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.926 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:20.926 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:20.926 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5795", 2024-04-24T14:35:20.926 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1567306440", 2024-04-24T14:35:20.926 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 56 2024-04-24T14:35:20.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.926 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399479+0000", 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399479+0000", 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399476+0000", 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.927 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399491+0000", 2024-04-24T14:35:20.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5688999999999998e-05 2024-04-24T14:35:20.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399527+0000", 2024-04-24T14:35:20.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6192999999999997e-05 2024-04-24T14:35:20.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.928 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.928 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.928 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399547+0000", 2024-04-24T14:35:20.928 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9156000000000001e-05 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399650+0000", 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103346 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399663+0000", 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.345e-05 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.402177+0000", 2024-04-24T14:35:20.929 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025136659999999999 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.402336+0000", 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015912700000000001 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.402739+0000", 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000402736 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.413101+0000", 2024-04-24T14:35:20.930 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010361746 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.413152+0000", 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1515999999999999e-05 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.413206+0000", 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4316e-05 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.413248+0000", 2024-04-24T14:35:20.931 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1820000000000003e-05 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.2.0:212 3.0 e116/99 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:7 v 116'272, mlcod=115'266)", 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:17.401237+0000", 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 3.4488038689999998, 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011174969999999999, 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:20.932 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.401237+0000", 2024-04-24T14:35:20.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:20.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.401237+0000", 2024-04-24T14:35:20.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:20.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.933 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.933 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:20.933 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.401235+0000", 2024-04-24T14:35:20.933 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:20.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.401276+0000", 2024-04-24T14:35:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1009e-05 2024-04-24T14:35:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.401321+0000", 2024-04-24T14:35:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4778999999999999e-05 2024-04-24T14:35:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:20.934 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.401337+0000", 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5807e-05 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.403013+0000", 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016755909999999999 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.403037+0000", 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4485e-05 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:20.935 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.412346+0000", 2024-04-24T14:35:20.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093090180000000005 2024-04-24T14:35:20.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:20.936 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:20.936 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:20.936 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.412412+0000", 2024-04-24T14:35:20.936 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5983999999999997e-05 2024-04-24T14:35:20.936 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.936 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.936 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.936 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:20.936 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:20.936 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:20.938 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:35:20.979 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:20.984+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:21.044 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-04-24T14:35:21.046+0000 7ff1891d6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 20303) UID: 0 2024-04-24T14:35:21.082 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:21.083 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:21.083 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:35:21.083 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:21.092 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:35:21.145 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:21.146+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:21.191 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:21.192 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:21.192 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:35:21.192 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:35:21.192 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:21.201 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:35:21.246 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:21.250+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:21.301 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:21.301 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:35:21.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:35:21.301 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:35:21.301 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.301 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:35:21.301 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:21.301 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 101.23255988299999, 2024-04-24T14:35:21.301 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:35:21.301 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.301 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.301 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:21.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.303 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:35:21.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:35:21.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:35:21.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:35:21.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.304 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.304 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:21.304 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:35:21.304 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:35:21.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.237745304000001, 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.305 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:21.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:21.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.306 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.306 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.306 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:35:21.306 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:35:21.307 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:35:21.308 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.226816045999996, 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.309 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:35:21.310 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.311 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:35:21.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:35:21.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:35:21.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:35:21.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.312 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.312 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:21.312 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:35:21.312 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:35:21.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.215859965000007, 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.313 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.314 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.315 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.316 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 87.204290599999993, 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.317 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:21.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:21.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.318 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.318 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:35:21.318 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:21.318 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.318 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.319 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:35:21.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:35:21.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:35:21.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:35:21.320 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.320 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.320 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:21.320 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:35:21.320 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:35:21.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.321 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.321 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:35:21.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:35:21.321 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.321 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.321 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.321 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.321 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:35:21.321 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:21.321 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 86.951170941000001, 2024-04-24T14:35:21.321 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:35:21.322 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.322 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.322 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:21.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.322 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.322 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:21.322 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:35:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:35:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:35:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:35:21.323 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:35:21.324 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:35:21.325 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:21.326 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 86.940425773000001, 2024-04-24T14:35:21.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:35:21.326 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.326 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.326 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:21.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.326 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.326 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:21.326 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:35:21.327 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:35:21.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.329 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:35:21.330 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:21.330 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 86.928059497000007, 2024-04-24T14:35:21.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:35:21.330 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.330 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.330 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:21.330 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.330 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.330 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:21.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:35:21.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:21.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.331 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.331 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:35:21.331 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:35:21.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.331 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.332 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.333 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 86.774007690000005, 2024-04-24T14:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.335 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.336 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.337 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:35:21.338 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:21.338 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 86.604169014000007, 2024-04-24T14:35:21.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:35:21.338 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.338 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.338 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:21.338 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.338 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.338 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.338 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:35:21.339 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:35:21.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:35:21.341 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:21.342 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 82.269198179, 2024-04-24T14:35:21.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:35:21.342 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.342 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.342 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:21.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.342 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.342 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.342 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:21.342 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:35:21.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:35:21.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:21.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:35:21.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:35:21.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.345 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.345 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.345 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:35:21.345 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:35:21.345 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.345 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.345 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.346 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:35:21.346 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:21.346 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 78.425047430999996, 2024-04-24T14:35:21.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:35:21.346 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.346 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:35:21.346 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.346 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.346 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:21.346 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.346 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:21.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.347 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.347 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:21.347 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:21.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.347 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:35:21.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:35:21.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:35:21.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:35:21.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.348 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.348 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.348 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:35:21.348 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.349 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.350 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:35:21.350 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:21.350 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 63.201284430999998, 2024-04-24T14:35:21.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044084859999999997, 2024-04-24T14:35:21.350 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.350 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:21.350 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.350 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.350 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:21.350 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.350 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:21.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103206+0000", 2024-04-24T14:35:21.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:21.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.351 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.351 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.351 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104731+0000", 2024-04-24T14:35:21.351 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001524755 2024-04-24T14:35:21.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104759+0000", 2024-04-24T14:35:21.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7973000000000001e-05 2024-04-24T14:35:21.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104807+0000", 2024-04-24T14:35:21.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8112000000000003e-05 2024-04-24T14:35:21.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.352 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.352 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.352 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104907+0000", 2024-04-24T14:35:21.352 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100317 2024-04-24T14:35:21.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104918+0000", 2024-04-24T14:35:21.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747999999999999e-05 2024-04-24T14:35:21.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.353 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.353 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.107616+0000", 2024-04-24T14:35:21.353 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026977559999999999 2024-04-24T14:35:21.353 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.353 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.353 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.354 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:35:21.354 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:21.354 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 63.177989478000001, 2024-04-24T14:35:21.354 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046193859999999996, 2024-04-24T14:35:21.354 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.354 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:21.354 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.354 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.354 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.354 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:21.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:21.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.355 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.355 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126501+0000", 2024-04-24T14:35:21.355 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:21.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.355 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128169+0000", 2024-04-24T14:35:21.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016686229999999999 2024-04-24T14:35:21.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128202+0000", 2024-04-24T14:35:21.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2539000000000001e-05 2024-04-24T14:35:21.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.356 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.356 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.356 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128222+0000", 2024-04-24T14:35:21.356 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0149e-05 2024-04-24T14:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128375+0000", 2024-04-24T14:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000153459 2024-04-24T14:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128399+0000", 2024-04-24T14:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3393000000000001e-05 2024-04-24T14:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.357 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.131122+0000", 2024-04-24T14:35:21.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027227140000000002 2024-04-24T14:35:21.358 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.358 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.358 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.358 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.358 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5674.0:27 3.2 3:5592b7cf:::journal.12ff64619e2d:head [watch watch cookie 139908626846656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T14:35:21.358 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.218271+0000", 2024-04-24T14:35:21.358 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 17.086220884999999, 2024-04-24T14:35:21.358 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.107948372, 2024-04-24T14:35:21.358 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.358 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.358 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5674", 2024-04-24T14:35:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1498157813", 2024-04-24T14:35:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 27 2024-04-24T14:35:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218271+0000", 2024-04-24T14:35:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218271+0000", 2024-04-24T14:35:21.359 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218267+0000", 2024-04-24T14:35:21.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:35:21.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218280+0000", 2024-04-24T14:35:21.360 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2469e-05 2024-04-24T14:35:21.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.360 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.360 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.360 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218303+0000", 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2952999999999999e-05 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218316+0000", 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.313e-05 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218446+0000", 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000130154 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218460+0000", 2024-04-24T14:35:21.361 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3852999999999999e-05 2024-04-24T14:35:21.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.320512+0000", 2024-04-24T14:35:21.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.102051709 2024-04-24T14:35:21.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.362 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for missing object", 2024-04-24T14:35:21.362 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.320660+0000", 2024-04-24T14:35:21.362 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000148362 2024-04-24T14:35:21.362 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.362 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.322855+0000", 2024-04-24T14:35:21.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021955249999999998 2024-04-24T14:35:21.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:21.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.322976+0000", 2024-04-24T14:35:21.363 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000120813 2024-04-24T14:35:21.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.363 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.363 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.323702+0000", 2024-04-24T14:35:21.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00072575199999999997 2024-04-24T14:35:21.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.323827+0000", 2024-04-24T14:35:21.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012478200000000001 2024-04-24T14:35:21.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.364 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:21.364 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.324040+0000", 2024-04-24T14:35:21.364 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021278700000000001 2024-04-24T14:35:21.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.364 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:21.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.324936+0000", 2024-04-24T14:35:21.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00089639600000000004 2024-04-24T14:35:21.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:21.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.326067+0000", 2024-04-24T14:35:21.365 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011309110000000001 2024-04-24T14:35:21.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.365 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.365 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:21.365 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.326114+0000", 2024-04-24T14:35:21.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6903999999999997e-05 2024-04-24T14:35:21.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.366 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.366 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.326219+0000", 2024-04-24T14:35:21.366 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105368 2024-04-24T14:35:21.366 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.366 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.366 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.366 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.366 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 108/107 [PushOp(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 49'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@49'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:35:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.396664+0000", 2024-04-24T14:35:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.907827966999999, 2024-04-24T14:35:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004962241, 2024-04-24T14:35:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.396664+0000", 2024-04-24T14:35:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.367 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.396664+0000", 2024-04-24T14:35:21.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.396662+0000", 2024-04-24T14:35:21.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:21.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.368 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.398018+0000", 2024-04-24T14:35:21.368 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001355904 2024-04-24T14:35:21.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.368 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.368 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.398037+0000", 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8907e-05 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.398049+0000", 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1661000000000001e-05 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.400590+0000", 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025413749999999998 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.369 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.400599+0000", 2024-04-24T14:35:21.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0420000000000008e-06 2024-04-24T14:35:21.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.370 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.370 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.401626+0000", 2024-04-24T14:35:21.370 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0010267119999999999 2024-04-24T14:35:21.370 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.370 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.370 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.370 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:8 2.2 2:4f9137c5:::journal_data.2.12ff64619e2d.3:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e108)", 2024-04-24T14:35:21.370 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.439647+0000", 2024-04-24T14:35:21.370 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.864844327, 2024-04-24T14:35:21.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050746209999999996, 2024-04-24T14:35:21.371 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.371 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:21.371 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439647+0000", 2024-04-24T14:35:21.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.371 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.371 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.371 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439647+0000", 2024-04-24T14:35:21.371 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439646+0000", 2024-04-24T14:35:21.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:21.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439652+0000", 2024-04-24T14:35:21.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0170000000000004e-06 2024-04-24T14:35:21.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.372 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.372 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.372 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439660+0000", 2024-04-24T14:35:21.372 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7730000000000005e-06 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439665+0000", 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5220000000000001e-06 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.441882+0000", 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022169260000000001 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.442038+0000", 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000156661 2024-04-24T14:35:21.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.443062+0000", 2024-04-24T14:35:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001023604 2024-04-24T14:35:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.443281+0000", 2024-04-24T14:35:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000218564 2024-04-24T14:35:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.444722+0000", 2024-04-24T14:35:21.374 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0014413539999999999 2024-04-24T14:35:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5674.0:200 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 3895296~36864] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-04-24T14:35:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.470076+0000", 2024-04-24T14:35:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.834415966000002, 2024-04-24T14:35:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044721860000000004, 2024-04-24T14:35:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5674", 2024-04-24T14:35:21.375 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1498157813", 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 200 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470076+0000", 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470076+0000", 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.376 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470075+0000", 2024-04-24T14:35:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470078+0000", 2024-04-24T14:35:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5459999999999998e-06 2024-04-24T14:35:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470083+0000", 2024-04-24T14:35:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2009999999999998e-06 2024-04-24T14:35:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.377 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470088+0000", 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7480000000000001e-06 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473569+0000", 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034816399999999998 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473887+0000", 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00031747200000000001 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.378 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.474548+0000", 2024-04-24T14:35:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066107099999999995 2024-04-24T14:35:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 108/107 [PushOp(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 49'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@49'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:35:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.470456+0000", 2024-04-24T14:35:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.834035932999999, 2024-04-24T14:35:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055876789999999999, 2024-04-24T14:35:21.379 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470456+0000", 2024-04-24T14:35:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.380 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470456+0000", 2024-04-24T14:35:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470454+0000", 2024-04-24T14:35:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470992+0000", 2024-04-24T14:35:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053743599999999997 2024-04-24T14:35:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.381 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.471024+0000", 2024-04-24T14:35:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2480000000000001e-05 2024-04-24T14:35:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.471045+0000", 2024-04-24T14:35:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0503e-05 2024-04-24T14:35:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473545+0000", 2024-04-24T14:35:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025000299999999999 2024-04-24T14:35:21.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473565+0000", 2024-04-24T14:35:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0352000000000001e-05 2024-04-24T14:35:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.476043+0000", 2024-04-24T14:35:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024783840000000001 2024-04-24T14:35:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5606.0:3 3.2 3:5592b7cf:::journal.12ff64619e2d:head [watch watch cookie 94231437332288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.527849+0000", 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 16.776642375000002, 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044396410000000002, 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5606", 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2386300588", 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 3 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:21.384 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527849+0000", 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527849+0000", 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527848+0000", 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527855+0000", 2024-04-24T14:35:21.385 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8719999999999996e-06 2024-04-24T14:35:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527890+0000", 2024-04-24T14:35:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4836999999999999e-05 2024-04-24T14:35:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527906+0000", 2024-04-24T14:35:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6535999999999999e-05 2024-04-24T14:35:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:21.386 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.529465+0000", 2024-04-24T14:35:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001559429 2024-04-24T14:35:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.529800+0000", 2024-04-24T14:35:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000334638 2024-04-24T14:35:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.530413+0000", 2024-04-24T14:35:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061266800000000002 2024-04-24T14:35:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.387 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532044+0000", 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001631565 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532061+0000", 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6874000000000001e-05 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532139+0000", 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8191999999999999e-05 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:21.388 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532289+0000", 2024-04-24T14:35:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014948199999999999 2024-04-24T14:35:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:21.389 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:21.389 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:21.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:35:21.391 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-04-24T14:35:21.394+0000 7f6701356700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 20298) UID: 0 2024-04-24T14:35:21.491 INFO:tasks.ceph.osd.3.smithi103.stderr:2024-04-24T14:35:21.496+0000 7f7139f56700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 20310) UID: 0 2024-04-24T14:35:21.494 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:21.494 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:21.494 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:35:21.494 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:21.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:35:21.518 INFO:tasks.workunit.client.0.smithi173.stderr:pool 'new_default_pool16809' does not exist 2024-04-24T14:35:21.528 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-24T14:35:21.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-24T14:35:21.539 INFO:tasks.workunit:Stopping ['rbd/rbd-nbd.sh'] on client.0... 2024-04-24T14:35:21.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-24T14:35:21.591 INFO:tasks.ceph.osd.5.smithi103.stderr:2024-04-24T14:35:21.596+0000 7f54841d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 20314) UID: 0 2024-04-24T14:35:21.600 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:21.600 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:21.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:35:21.691 INFO:tasks.ceph.osd.4.smithi103.stderr:2024-04-24T14:35:21.696+0000 7fe8633b6700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 20312) UID: 0 2024-04-24T14:35:21.709 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.834640023000006, 2024-04-24T14:35:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:35:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:35:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:35:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:35:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:35:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:35:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:35:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:35:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:35:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:35:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:35:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:35:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:35:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:35:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:35:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:35:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:35:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.834611038000006, 2024-04-24T14:35:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:35:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:35:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:35:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:35:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:35:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:35:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:35:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:35:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:35:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:35:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:35:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:35:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:35:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:35:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:35:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:35:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:35:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:35:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:35:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:35:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:35:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:35:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.834592552999993, 2024-04-24T14:35:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:35:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:35:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:35:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:35:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:35:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:35:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:35:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:35:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:35:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:35:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:35:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:35:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:35:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:35:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:35:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:35:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:35:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:35:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:35:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:35:21.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:35:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:35:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:35:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:21.727 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.834577272000004, 2024-04-24T14:35:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:35:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:35:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:35:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:35:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:35:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:35:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:35:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:35:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:35:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:35:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:35:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:35:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:35:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:35:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:35:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:35:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:35:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:35:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:35:21.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:35:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:35:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:35:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:21.733 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.834561491000002, 2024-04-24T14:35:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:35:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:35:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:35:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:35:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:35:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:35:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:35:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:35:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:35:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:35:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:35:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:35:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:35:21.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:35:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:35:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:35:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:35:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:35:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.834547203, 2024-04-24T14:35:21.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:35:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:35:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:35:21.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:35:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:35:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:35:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:35:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:35:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:35:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:35:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:35:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:35:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:35:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:35:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:35:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:35:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:35:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:35:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:35:21.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:35:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:35:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:35:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.834527451, 2024-04-24T14:35:21.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:35:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:35:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:21.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:35:21.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:35:21.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:35:21.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:35:21.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:35:21.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:35:21.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:35:21.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:35:21.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:21.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:35:21.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:35:21.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:35:21.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:35:21.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:21.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:35:21.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:35:21.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:35:21.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:35:21.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:35:21.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:35:21.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:35:21.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:35:21.751 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.751 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.751 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.751 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:21.751 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:21.751 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.834510003999995, 2024-04-24T14:35:21.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:35:21.752 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.752 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:21.752 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.752 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:21.752 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:21.752 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:35:21.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.752 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:21.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:21.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:21.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:21.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:35:21.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:35:21.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:35:21.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:35:21.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:35:21.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:35:21.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:35:21.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:35:21.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:21.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:35:21.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:35:21.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:35:21.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:35:21.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:21.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:35:21.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:35:21.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:35:21.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:35:21.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:35:21.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:35:21.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:35:21.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:35:21.757 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.757 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.757 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.757 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:100 3.2 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:21.757 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:21.758 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.960455189000001, 2024-04-24T14:35:21.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009078015, 2024-04-24T14:35:21.758 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.758 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.758 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.758 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:21.758 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:21.758 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 100 2024-04-24T14:35:21.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.758 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:21.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:21.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754019+0000", 2024-04-24T14:35:21.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:21.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754025+0000", 2024-04-24T14:35:21.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2110000000000003e-06 2024-04-24T14:35:21.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754037+0000", 2024-04-24T14:35:21.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1868999999999999e-05 2024-04-24T14:35:21.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754046+0000", 2024-04-24T14:35:21.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7430000000000008e-06 2024-04-24T14:35:21.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754095+0000", 2024-04-24T14:35:21.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8924000000000001e-05 2024-04-24T14:35:21.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754153+0000", 2024-04-24T14:35:21.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8165000000000003e-05 2024-04-24T14:35:21.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:21.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754262+0000", 2024-04-24T14:35:21.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108694 2024-04-24T14:35:21.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:21.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.762999+0000", 2024-04-24T14:35:21.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087372590000000007 2024-04-24T14:35:21.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:21.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763027+0000", 2024-04-24T14:35:21.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8467e-05 2024-04-24T14:35:21.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:21.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763070+0000", 2024-04-24T14:35:21.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2236e-05 2024-04-24T14:35:21.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763098+0000", 2024-04-24T14:35:21.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8561999999999999e-05 2024-04-24T14:35:21.763 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.763 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.763 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.763 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:101 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [delete] snapc 7=[7,6,4] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:21.764 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:21.764 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.755247523000001, 2024-04-24T14:35:21.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012894743, 2024-04-24T14:35:21.764 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.764 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.764 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.764 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:21.764 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:21.764 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 101 2024-04-24T14:35:21.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.764 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:21.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:21.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959227+0000", 2024-04-24T14:35:21.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:21.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959233+0000", 2024-04-24T14:35:21.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.857e-06 2024-04-24T14:35:21.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959249+0000", 2024-04-24T14:35:21.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6155999999999999e-05 2024-04-24T14:35:21.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959257+0000", 2024-04-24T14:35:21.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7440000000000007e-06 2024-04-24T14:35:21.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959790+0000", 2024-04-24T14:35:21.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053381100000000003 2024-04-24T14:35:21.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960206+0000", 2024-04-24T14:35:21.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041549700000000002 2024-04-24T14:35:21.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:21.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960744+0000", 2024-04-24T14:35:21.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053762199999999997 2024-04-24T14:35:21.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:21.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971963+0000", 2024-04-24T14:35:21.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011219816000000001 2024-04-24T14:35:21.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:21.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972021+0000", 2024-04-24T14:35:21.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7855000000000001e-05 2024-04-24T14:35:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972073+0000", 2024-04-24T14:35:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2196e-05 2024-04-24T14:35:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972123+0000", 2024-04-24T14:35:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9311999999999998e-05 2024-04-24T14:35:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.770 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:102 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:21.770 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:21.770 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.755207497000001, 2024-04-24T14:35:21.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012939713, 2024-04-24T14:35:21.770 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.770 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:21.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:21.770 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 102 2024-04-24T14:35:21.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.770 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959267+0000", 2024-04-24T14:35:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959274+0000", 2024-04-24T14:35:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.593e-06 2024-04-24T14:35:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959291+0000", 2024-04-24T14:35:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6387000000000001e-05 2024-04-24T14:35:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959301+0000", 2024-04-24T14:35:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0023e-05 2024-04-24T14:35:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969594+0000", 2024-04-24T14:35:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010293182 2024-04-24T14:35:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969902+0000", 2024-04-24T14:35:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030835299999999998 2024-04-24T14:35:21.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970408+0000", 2024-04-24T14:35:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050588100000000002 2024-04-24T14:35:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972009+0000", 2024-04-24T14:35:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016011210000000001 2024-04-24T14:35:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972132+0000", 2024-04-24T14:35:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122521 2024-04-24T14:35:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972174+0000", 2024-04-24T14:35:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2242e-05 2024-04-24T14:35:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972208+0000", 2024-04-24T14:35:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3522999999999997e-05 2024-04-24T14:35:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.775 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:104 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.754596282999998, 2024-04-24T14:35:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014589062, 2024-04-24T14:35:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 104 2024-04-24T14:35:21.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959877+0000", 2024-04-24T14:35:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959890+0000", 2024-04-24T14:35:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2537e-05 2024-04-24T14:35:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959909+0000", 2024-04-24T14:35:21.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9340000000000001e-05 2024-04-24T14:35:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959922+0000", 2024-04-24T14:35:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2882e-05 2024-04-24T14:35:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970900+0000", 2024-04-24T14:35:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010977736 2024-04-24T14:35:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971039+0000", 2024-04-24T14:35:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139593 2024-04-24T14:35:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971453+0000", 2024-04-24T14:35:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041351099999999998 2024-04-24T14:35:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974044+0000", 2024-04-24T14:35:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025908289999999998 2024-04-24T14:35:21.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974377+0000", 2024-04-24T14:35:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033310399999999997 2024-04-24T14:35:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974433+0000", 2024-04-24T14:35:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6166999999999999e-05 2024-04-24T14:35:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974468+0000", 2024-04-24T14:35:21.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5336000000000002e-05 2024-04-24T14:35:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:106 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.754536950000002, 2024-04-24T14:35:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017192064999999999, 2024-04-24T14:35:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 106 2024-04-24T14:35:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959938+0000", 2024-04-24T14:35:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959943+0000", 2024-04-24T14:35:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.677e-06 2024-04-24T14:35:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959953+0000", 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8069999999999994e-06 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0560000000000001e-06 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972232+0000", 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012272460000000001 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972304+0000", 2024-04-24T14:35:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2630000000000001e-05 2024-04-24T14:35:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972548+0000", 2024-04-24T14:35:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000244047 2024-04-24T14:35:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976900+0000", 2024-04-24T14:35:21.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043514069999999998 2024-04-24T14:35:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977076+0000", 2024-04-24T14:35:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176281 2024-04-24T14:35:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977106+0000", 2024-04-24T14:35:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0015e-05 2024-04-24T14:35:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977131+0000", 2024-04-24T14:35:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4669e-05 2024-04-24T14:35:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:114 3.2 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.731215786, 2024-04-24T14:35:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090880089999999993, 2024-04-24T14:35:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 114 2024-04-24T14:35:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983259+0000", 2024-04-24T14:35:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983264+0000", 2024-04-24T14:35:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9130000000000002e-06 2024-04-24T14:35:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983277+0000", 2024-04-24T14:35:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3723999999999999e-05 2024-04-24T14:35:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983282+0000", 2024-04-24T14:35:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0429999999999998e-06 2024-04-24T14:35:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983339+0000", 2024-04-24T14:35:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6323000000000003e-05 2024-04-24T14:35:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983464+0000", 2024-04-24T14:35:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012554299999999999 2024-04-24T14:35:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983693+0000", 2024-04-24T14:35:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000228848 2024-04-24T14:35:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.986978+0000", 2024-04-24T14:35:21.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032852810000000001 2024-04-24T14:35:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992288+0000", 2024-04-24T14:35:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053095369999999996 2024-04-24T14:35:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992322+0000", 2024-04-24T14:35:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3683999999999998e-05 2024-04-24T14:35:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992348+0000", 2024-04-24T14:35:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6157999999999999e-05 2024-04-24T14:35:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:121 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.719159687000001, 2024-04-24T14:35:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098798159999999996, 2024-04-24T14:35:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 121 2024-04-24T14:35:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995315+0000", 2024-04-24T14:35:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995320+0000", 2024-04-24T14:35:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0239999999999997e-06 2024-04-24T14:35:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995328+0000", 2024-04-24T14:35:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7800000000000001e-06 2024-04-24T14:35:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995334+0000", 2024-04-24T14:35:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2330000000000003e-06 2024-04-24T14:35:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998810+0000", 2024-04-24T14:35:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034756090000000002 2024-04-24T14:35:21.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998870+0000", 2024-04-24T14:35:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0625999999999999e-05 2024-04-24T14:35:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.999091+0000", 2024-04-24T14:35:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000220631 2024-04-24T14:35:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:21.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.001389+0000", 2024-04-24T14:35:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022977549999999998 2024-04-24T14:35:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005097+0000", 2024-04-24T14:35:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037085439999999998 2024-04-24T14:35:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005154+0000", 2024-04-24T14:35:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6208000000000001e-05 2024-04-24T14:35:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005196+0000", 2024-04-24T14:35:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2052000000000003e-05 2024-04-24T14:35:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:127 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.708858171000003, 2024-04-24T14:35:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013414918, 2024-04-24T14:35:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.800 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2024-04-24T14:35:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005616+0000", 2024-04-24T14:35:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005622+0000", 2024-04-24T14:35:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2020000000000003e-06 2024-04-24T14:35:21.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005632+0000", 2024-04-24T14:35:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2024-04-24T14:35:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005637+0000", 2024-04-24T14:35:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7439999999999998e-06 2024-04-24T14:35:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005947+0000", 2024-04-24T14:35:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000310792 2024-04-24T14:35:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006059+0000", 2024-04-24T14:35:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011192399999999999 2024-04-24T14:35:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006503+0000", 2024-04-24T14:35:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000443394 2024-04-24T14:35:21.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018864+0000", 2024-04-24T14:35:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012361162 2024-04-24T14:35:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018877+0000", 2024-04-24T14:35:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3497e-05 2024-04-24T14:35:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:21.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018965+0000", 2024-04-24T14:35:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7476000000000001e-05 2024-04-24T14:35:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019032+0000", 2024-04-24T14:35:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7469000000000001e-05 2024-04-24T14:35:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:129 3.2 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:21.806 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.708717937999999, 2024-04-24T14:35:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010059754000000001, 2024-04-24T14:35:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 129 2024-04-24T14:35:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:21.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005757+0000", 2024-04-24T14:35:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005760+0000", 2024-04-24T14:35:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2430000000000001e-06 2024-04-24T14:35:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005769+0000", 2024-04-24T14:35:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4810000000000006e-06 2024-04-24T14:35:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005773+0000", 2024-04-24T14:35:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:35:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005840+0000", 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6934999999999995e-05 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005961+0000", 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012153 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006181+0000", 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021947500000000001 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015520+0000", 2024-04-24T14:35:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093393049999999991 2024-04-24T14:35:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015733+0000", 2024-04-24T14:35:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021355699999999999 2024-04-24T14:35:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015773+0000", 2024-04-24T14:35:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9554000000000001e-05 2024-04-24T14:35:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015817+0000", 2024-04-24T14:35:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4325e-05 2024-04-24T14:35:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:131 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.708440056999997, 2024-04-24T14:35:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014251976, 2024-04-24T14:35:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.812 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:21.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006034+0000", 2024-04-24T14:35:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006042+0000", 2024-04-24T14:35:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2970000000000004e-06 2024-04-24T14:35:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006061+0000", 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8505999999999999e-05 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006071+0000", 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0409e-05 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007899+0000", 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018274789999999999 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008281+0000", 2024-04-24T14:35:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038244700000000001 2024-04-24T14:35:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008528+0000", 2024-04-24T14:35:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024723900000000001 2024-04-24T14:35:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019070+0000", 2024-04-24T14:35:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010541867999999999 2024-04-24T14:35:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020099+0000", 2024-04-24T14:35:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00102852 2024-04-24T14:35:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020226+0000", 2024-04-24T14:35:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012744300000000001 2024-04-24T14:35:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020287+0000", 2024-04-24T14:35:21.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1253000000000005e-05 2024-04-24T14:35:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:132 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.708386204, 2024-04-24T14:35:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015917981000000001, 2024-04-24T14:35:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:21.818 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 132 2024-04-24T14:35:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006093+0000", 2024-04-24T14:35:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3819999999999997e-06 2024-04-24T14:35:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006103+0000", 2024-04-24T14:35:21.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0352e-05 2024-04-24T14:35:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006110+0000", 2024-04-24T14:35:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3910000000000003e-06 2024-04-24T14:35:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015586+0000", 2024-04-24T14:35:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094759119999999995 2024-04-24T14:35:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015730+0000", 2024-04-24T14:35:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014487199999999999 2024-04-24T14:35:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015886+0000", 2024-04-24T14:35:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015573799999999999 2024-04-24T14:35:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020306+0000", 2024-04-24T14:35:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044199399999999998 2024-04-24T14:35:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:21.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021887+0000", 2024-04-24T14:35:21.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015809400000000001 2024-04-24T14:35:21.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:21.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021942+0000", 2024-04-24T14:35:21.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5071000000000002e-05 2024-04-24T14:35:21.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.022007+0000", 2024-04-24T14:35:21.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5110000000000005e-05 2024-04-24T14:35:21.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.824 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.824 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.824 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.5324.0:127 3.0 e87/69 ondisk, result = 0)", 2024-04-24T14:35:21.824 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:21.824 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.706677136000003, 2024-04-24T14:35:21.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011245125, 2024-04-24T14:35:21.824 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:21.824 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:21.824 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:21.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:21.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007804+0000", 2024-04-24T14:35:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6610000000000003e-06 2024-04-24T14:35:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007817+0000", 2024-04-24T14:35:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3030999999999999e-05 2024-04-24T14:35:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007832+0000", 2024-04-24T14:35:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.47e-05 2024-04-24T14:35:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018843+0000", 2024-04-24T14:35:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011010647 2024-04-24T14:35:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018853+0000", 2024-04-24T14:35:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319000000000001e-05 2024-04-24T14:35:21.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019043+0000", 2024-04-24T14:35:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019055600000000001 2024-04-24T14:35:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:21.828 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:21.828 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:21.830 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-04-24T14:35:21.834+0000 7f229abcb700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 34408) UID: 0 2024-04-24T14:35:21.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-24T14:35:21.831 DEBUG:teuthology.parallel:result is None 2024-04-24T14:35:21.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-24T14:35:21.846 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-24T14:35:21.847 DEBUG:teuthology.orchestra.run.smithi173:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-24T14:35:21.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-24T14:35:21.906 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-24T14:35:21.906 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-24T14:35:21.916 INFO:tasks.thrashosds:joining thrashosds 2024-04-24T14:35:21.941 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:21.941 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:21.941 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:35:21.941 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:21.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-24T14:35:22.041 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:22.041 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:22.041 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:22.041 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:22.041 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:22.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-24T14:35:22.164 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:22.164 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:22.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:22.164 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:22.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.164 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:467 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.41306019300001, 2024-04-24T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034219656000000001, 2024-04-24T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 467 2024-04-24T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756305+0000", 2024-04-24T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756304+0000", 2024-04-24T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756307+0000", 2024-04-24T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0149999999999999e-06 2024-04-24T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756312+0000", 2024-04-24T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.848e-06 2024-04-24T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756315+0000", 2024-04-24T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7499999999999999e-06 2024-04-24T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756822+0000", 2024-04-24T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050680899999999999 2024-04-24T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756866+0000", 2024-04-24T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4246000000000003e-05 2024-04-24T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778740+0000", 2024-04-24T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.021874451 2024-04-24T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.778973+0000", 2024-04-24T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023283200000000001 2024-04-24T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789597+0000", 2024-04-24T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010624122 2024-04-24T14:35:22.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.789747+0000", 2024-04-24T14:35:22.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014982099999999999 2024-04-24T14:35:22.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790524+0000", 2024-04-24T14:35:22.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077720399999999998 2024-04-24T14:35:22.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.170 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:468 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.413009359, 2024-04-24T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035441367000000001, 2024-04-24T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2024-04-24T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756355+0000", 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756362+0000", 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4649999999999999e-06 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756375+0000", 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3730000000000001e-05 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756380+0000", 2024-04-24T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1830000000000002e-06 2024-04-24T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756929+0000", 2024-04-24T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054851499999999996 2024-04-24T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756985+0000", 2024-04-24T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5954e-05 2024-04-24T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779028+0000", 2024-04-24T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022042626999999999 2024-04-24T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779113+0000", 2024-04-24T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5852000000000004e-05 2024-04-24T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790670+0000", 2024-04-24T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011556345000000001 2024-04-24T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.790907+0000", 2024-04-24T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023677999999999999 2024-04-24T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791797+0000", 2024-04-24T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089027799999999999 2024-04-24T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:469 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.412973052, 2024-04-24T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036236890000000001, 2024-04-24T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2024-04-24T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756392+0000", 2024-04-24T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756391+0000", 2024-04-24T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756394+0000", 2024-04-24T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.21e-06 2024-04-24T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756401+0000", 2024-04-24T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.878e-06 2024-04-24T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.756403+0000", 2024-04-24T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.791e-06 2024-04-24T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757020+0000", 2024-04-24T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061705400000000002 2024-04-24T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.757063+0000", 2024-04-24T14:35:22.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2385999999999997e-05 2024-04-24T14:35:22.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779187+0000", 2024-04-24T14:35:22.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022124403000000001 2024-04-24T14:35:22.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779384+0000", 2024-04-24T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019709099999999999 2024-04-24T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791834+0000", 2024-04-24T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012449511 2024-04-24T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.791948+0000", 2024-04-24T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114133 2024-04-24T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792629+0000", 2024-04-24T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068072399999999996 2024-04-24T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:470 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.409561802, 2024-04-24T14:35:22.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034098732999999999, 2024-04-24T14:35:22.184 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.184 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.184 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 470 2024-04-24T14:35:22.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.184 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:22.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759803+0000", 2024-04-24T14:35:22.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759801+0000", 2024-04-24T14:35:22.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:22.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759809+0000", 2024-04-24T14:35:22.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7360000000000001e-06 2024-04-24T14:35:22.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759827+0000", 2024-04-24T14:35:22.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7583000000000001e-05 2024-04-24T14:35:22.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759837+0000", 2024-04-24T14:35:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9699999999999994e-06 2024-04-24T14:35:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759908+0000", 2024-04-24T14:35:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1524000000000002e-05 2024-04-24T14:35:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759973+0000", 2024-04-24T14:35:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4941e-05 2024-04-24T14:35:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779448+0000", 2024-04-24T14:35:22.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019474463000000001 2024-04-24T14:35:22.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779514+0000", 2024-04-24T14:35:22.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6072e-05 2024-04-24T14:35:22.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.792734+0000", 2024-04-24T14:35:22.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013220674999999999 2024-04-24T14:35:22.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793007+0000", 2024-04-24T14:35:22.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027285699999999999 2024-04-24T14:35:22.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793902+0000", 2024-04-24T14:35:22.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089444999999999998 2024-04-24T14:35:22.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.189 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:471 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.40950012499999, 2024-04-24T14:35:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.034974269000000002, 2024-04-24T14:35:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 471 2024-04-24T14:35:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.190 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:22.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759865+0000", 2024-04-24T14:35:22.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759864+0000", 2024-04-24T14:35:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759871+0000", 2024-04-24T14:35:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7419999999999997e-06 2024-04-24T14:35:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759883+0000", 2024-04-24T14:35:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1097999999999999e-05 2024-04-24T14:35:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759891+0000", 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0020000000000006e-06 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760019+0000", 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012799099999999999 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760047+0000", 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8101000000000001e-05 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779588+0000", 2024-04-24T14:35:22.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019541343999999999 2024-04-24T14:35:22.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779782+0000", 2024-04-24T14:35:22.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019443499999999999 2024-04-24T14:35:22.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.793993+0000", 2024-04-24T14:35:22.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014211044000000001 2024-04-24T14:35:22.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794109+0000", 2024-04-24T14:35:22.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115189 2024-04-24T14:35:22.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794839+0000", 2024-04-24T14:35:22.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073031099999999998 2024-04-24T14:35:22.195 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.196 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.196 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.196 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:472 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.196 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:22.196 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.409465549, 2024-04-24T14:35:22.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036053101999999997, 2024-04-24T14:35:22.196 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.196 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.196 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.196 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.196 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.196 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 472 2024-04-24T14:35:22.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.197 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:22.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:22.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759899+0000", 2024-04-24T14:35:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759903+0000", 2024-04-24T14:35:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2019999999999997e-06 2024-04-24T14:35:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759915+0000", 2024-04-24T14:35:22.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2367e-05 2024-04-24T14:35:22.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759921+0000", 2024-04-24T14:35:22.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.446e-06 2024-04-24T14:35:22.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760098+0000", 2024-04-24T14:35:22.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017766700000000001 2024-04-24T14:35:22.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760136+0000", 2024-04-24T14:35:22.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7302999999999998e-05 2024-04-24T14:35:22.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779859+0000", 2024-04-24T14:35:22.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019723725000000001 2024-04-24T14:35:22.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.779927+0000", 2024-04-24T14:35:22.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7562000000000001e-05 2024-04-24T14:35:22.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.794925+0000", 2024-04-24T14:35:22.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014998417 2024-04-24T14:35:22.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795123+0000", 2024-04-24T14:35:22.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019742100000000001 2024-04-24T14:35:22.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.795952+0000", 2024-04-24T14:35:22.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082965999999999997 2024-04-24T14:35:22.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.202 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.202 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:473 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.202 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:22.202 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.40943621, 2024-04-24T14:35:22.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037067983999999998, 2024-04-24T14:35:22.202 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.202 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.202 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.202 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.203 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.203 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 473 2024-04-24T14:35:22.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.203 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:22.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759929+0000", 2024-04-24T14:35:22.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759928+0000", 2024-04-24T14:35:22.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:22.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759932+0000", 2024-04-24T14:35:22.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7799999999999998e-06 2024-04-24T14:35:22.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759949+0000", 2024-04-24T14:35:22.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7842e-05 2024-04-24T14:35:22.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.759964+0000", 2024-04-24T14:35:22.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4535999999999999e-05 2024-04-24T14:35:22.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760182+0000", 2024-04-24T14:35:22.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021841699999999999 2024-04-24T14:35:22.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760225+0000", 2024-04-24T14:35:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2419999999999997e-05 2024-04-24T14:35:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780000+0000", 2024-04-24T14:35:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019774837999999999 2024-04-24T14:35:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780196+0000", 2024-04-24T14:35:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019606300000000001 2024-04-24T14:35:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796041+0000", 2024-04-24T14:35:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015845542000000001 2024-04-24T14:35:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796207+0000", 2024-04-24T14:35:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165699 2024-04-24T14:35:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.796997+0000", 2024-04-24T14:35:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000789669 2024-04-24T14:35:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:474 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.409359628, 2024-04-24T14:35:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.038046214000000002, 2024-04-24T14:35:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.209 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.209 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.209 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 474 2024-04-24T14:35:22.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.209 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:22.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760005+0000", 2024-04-24T14:35:22.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760004+0000", 2024-04-24T14:35:22.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:22.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760009+0000", 2024-04-24T14:35:22.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7879999999999997e-06 2024-04-24T14:35:22.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760020+0000", 2024-04-24T14:35:22.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2024-04-24T14:35:22.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760028+0000", 2024-04-24T14:35:22.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8620000000000003e-06 2024-04-24T14:35:22.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760272+0000", 2024-04-24T14:35:22.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024326999999999999 2024-04-24T14:35:22.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760314+0000", 2024-04-24T14:35:22.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2020000000000001e-05 2024-04-24T14:35:22.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780260+0000", 2024-04-24T14:35:22.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019946216999999999 2024-04-24T14:35:22.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780324+0000", 2024-04-24T14:35:22.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4109999999999994e-05 2024-04-24T14:35:22.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797077+0000", 2024-04-24T14:35:22.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016752843 2024-04-24T14:35:22.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.797317+0000", 2024-04-24T14:35:22.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000240132 2024-04-24T14:35:22.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798051+0000", 2024-04-24T14:35:22.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00073452700000000001 2024-04-24T14:35:22.214 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.214 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.214 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.214 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:475 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.214 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:22.214 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.409312083, 2024-04-24T14:35:22.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039040789999999999, 2024-04-24T14:35:22.214 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.215 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.215 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.215 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.215 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.215 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 475 2024-04-24T14:35:22.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.215 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:22.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760053+0000", 2024-04-24T14:35:22.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760052+0000", 2024-04-24T14:35:22.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:22.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760057+0000", 2024-04-24T14:35:22.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0139999999999998e-06 2024-04-24T14:35:22.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760068+0000", 2024-04-24T14:35:22.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1326e-05 2024-04-24T14:35:22.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760076+0000", 2024-04-24T14:35:22.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7139999999999993e-06 2024-04-24T14:35:22.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760354+0000", 2024-04-24T14:35:22.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027777200000000002 2024-04-24T14:35:22.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760390+0000", 2024-04-24T14:35:22.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6332999999999999e-05 2024-04-24T14:35:22.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780412+0000", 2024-04-24T14:35:22.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020022557 2024-04-24T14:35:22.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780609+0000", 2024-04-24T14:35:22.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632300000000001 2024-04-24T14:35:22.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798136+0000", 2024-04-24T14:35:22.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017527556 2024-04-24T14:35:22.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.798297+0000", 2024-04-24T14:35:22.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016021000000000001 2024-04-24T14:35:22.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799094+0000", 2024-04-24T14:35:22.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079695500000000002 2024-04-24T14:35:22.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.220 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.221 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:476 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.221 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:22.221 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.409271441, 2024-04-24T14:35:22.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040073316999999997, 2024-04-24T14:35:22.221 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.221 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.221 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.221 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.221 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.221 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 476 2024-04-24T14:35:22.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.221 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:22.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:22.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760093+0000", 2024-04-24T14:35:22.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:22.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760096+0000", 2024-04-24T14:35:22.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9000000000000002e-06 2024-04-24T14:35:22.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760102+0000", 2024-04-24T14:35:22.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5599999999999999e-06 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760106+0000", 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7560000000000001e-06 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760430+0000", 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032354699999999998 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760469+0000", 2024-04-24T14:35:22.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9731000000000003e-05 2024-04-24T14:35:22.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780679+0000", 2024-04-24T14:35:22.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02020943 2024-04-24T14:35:22.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780745+0000", 2024-04-24T14:35:22.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5782000000000001e-05 2024-04-24T14:35:22.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799178+0000", 2024-04-24T14:35:22.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018433307999999999 2024-04-24T14:35:22.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.799358+0000", 2024-04-24T14:35:22.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018027599999999999 2024-04-24T14:35:22.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800167+0000", 2024-04-24T14:35:22.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000808543 2024-04-24T14:35:22.227 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.227 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.227 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.227 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:477 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.227 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:22.227 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.40925433, 2024-04-24T14:35:22.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.041055810999999998, 2024-04-24T14:35:22.228 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.228 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.228 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.228 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.228 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.228 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 477 2024-04-24T14:35:22.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.228 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:22.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760111+0000", 2024-04-24T14:35:22.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760110+0000", 2024-04-24T14:35:22.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:22.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760113+0000", 2024-04-24T14:35:22.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7460000000000001e-06 2024-04-24T14:35:22.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760118+0000", 2024-04-24T14:35:22.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2379999999999997e-06 2024-04-24T14:35:22.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760122+0000", 2024-04-24T14:35:22.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6899999999999998e-06 2024-04-24T14:35:22.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760495+0000", 2024-04-24T14:35:22.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037345 2024-04-24T14:35:22.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.760539+0000", 2024-04-24T14:35:22.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3815e-05 2024-04-24T14:35:22.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.780798+0000", 2024-04-24T14:35:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020259152999999998 2024-04-24T14:35:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.781006+0000", 2024-04-24T14:35:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020787600000000001 2024-04-24T14:35:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800252+0000", 2024-04-24T14:35:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019245591999999999 2024-04-24T14:35:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.800402+0000", 2024-04-24T14:35:22.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015045600000000001 2024-04-24T14:35:22.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.801166+0000", 2024-04-24T14:35:22.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076433299999999996 2024-04-24T14:35:22.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.233 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.234 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:103 3.1 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:22.234 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:22.234 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.209688217, 2024-04-24T14:35:22.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032646351999999997, 2024-04-24T14:35:22.234 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.234 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:22.234 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.234 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:22.234 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:22.234 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 103 2024-04-24T14:35:22.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.234 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:22.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959677+0000", 2024-04-24T14:35:22.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959674+0000", 2024-04-24T14:35:22.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:22.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959688+0000", 2024-04-24T14:35:22.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2024-04-24T14:35:22.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959722+0000", 2024-04-24T14:35:22.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3936000000000002e-05 2024-04-24T14:35:22.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959735+0000", 2024-04-24T14:35:22.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2926e-05 2024-04-24T14:35:22.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959921+0000", 2024-04-24T14:35:22.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185601 2024-04-24T14:35:22.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960324+0000", 2024-04-24T14:35:22.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040356700000000002 2024-04-24T14:35:22.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:22.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960839+0000", 2024-04-24T14:35:22.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051410599999999996 2024-04-24T14:35:22.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:22.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.975634+0000", 2024-04-24T14:35:22.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014795770999999999 2024-04-24T14:35:22.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:22.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992113+0000", 2024-04-24T14:35:22.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016478982999999999 2024-04-24T14:35:22.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:22.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992228+0000", 2024-04-24T14:35:22.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114295 2024-04-24T14:35:22.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992323+0000", 2024-04-24T14:35:22.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5344000000000001e-05 2024-04-24T14:35:22.239 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.239 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.239 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.240 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:105 3.1 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:22.240 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:22.240 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.209438439000003, 2024-04-24T14:35:22.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032581757000000003, 2024-04-24T14:35:22.240 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.240 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:22.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:22.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:22.241 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 105 2024-04-24T14:35:22.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.241 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:22.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959926+0000", 2024-04-24T14:35:22.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959925+0000", 2024-04-24T14:35:22.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:22.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959937+0000", 2024-04-24T14:35:22.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2391999999999999e-05 2024-04-24T14:35:22.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:35:22.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2535999999999999e-05 2024-04-24T14:35:22.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959970+0000", 2024-04-24T14:35:22.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1039999999999999e-05 2024-04-24T14:35:22.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969531+0000", 2024-04-24T14:35:22.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095603949999999993 2024-04-24T14:35:22.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969669+0000", 2024-04-24T14:35:22.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013806499999999999 2024-04-24T14:35:22.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:22.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970010+0000", 2024-04-24T14:35:22.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034062200000000001 2024-04-24T14:35:22.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:22.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976983+0000", 2024-04-24T14:35:22.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069735600000000002 2024-04-24T14:35:22.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:22.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992413+0000", 2024-04-24T14:35:22.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015429841999999999 2024-04-24T14:35:22.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:22.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992457+0000", 2024-04-24T14:35:22.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4169999999999999e-05 2024-04-24T14:35:22.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992508+0000", 2024-04-24T14:35:22.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0995000000000002e-05 2024-04-24T14:35:22.246 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.246 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.246 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.247 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:113 3.1 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:22.247 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:22.247 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.189212148000003, 2024-04-24T14:35:22.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014793863000000001, 2024-04-24T14:35:22.247 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.247 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:22.247 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.247 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:22.247 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:22.247 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 113 2024-04-24T14:35:22.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.247 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:22.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980153+0000", 2024-04-24T14:35:22.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980152+0000", 2024-04-24T14:35:22.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:22.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980156+0000", 2024-04-24T14:35:22.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.619e-06 2024-04-24T14:35:22.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980163+0000", 2024-04-24T14:35:22.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7070000000000003e-06 2024-04-24T14:35:22.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.980173+0000", 2024-04-24T14:35:22.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0957e-05 2024-04-24T14:35:22.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983381+0000", 2024-04-24T14:35:22.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032079930000000001 2024-04-24T14:35:22.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983628+0000", 2024-04-24T14:35:22.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024665700000000001 2024-04-24T14:35:22.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:22.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983964+0000", 2024-04-24T14:35:22.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000336332 2024-04-24T14:35:22.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:22.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992694+0000", 2024-04-24T14:35:22.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087297009999999994 2024-04-24T14:35:22.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:22.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994878+0000", 2024-04-24T14:35:22.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021841439999999998 2024-04-24T14:35:22.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:22.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994917+0000", 2024-04-24T14:35:22.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8544999999999998e-05 2024-04-24T14:35:22.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.994947+0000", 2024-04-24T14:35:22.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9705000000000001e-05 2024-04-24T14:35:22.253 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.253 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.253 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.253 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:130 3.1 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:22.253 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:22.253 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.163518713000002, 2024-04-24T14:35:22.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014646289999999999, 2024-04-24T14:35:22.253 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.253 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:22.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:22.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:22.254 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 130 2024-04-24T14:35:22.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.254 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:22.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005846+0000", 2024-04-24T14:35:22.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005844+0000", 2024-04-24T14:35:22.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:22.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005853+0000", 2024-04-24T14:35:22.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1319999999999994e-06 2024-04-24T14:35:22.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005870+0000", 2024-04-24T14:35:22.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7448999999999998e-05 2024-04-24T14:35:22.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005876+0000", 2024-04-24T14:35:22.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3319999999999997e-06 2024-04-24T14:35:22.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005913+0000", 2024-04-24T14:35:22.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6835999999999998e-05 2024-04-24T14:35:22.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006017+0000", 2024-04-24T14:35:22.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010358500000000001 2024-04-24T14:35:22.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:22.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006139+0000", 2024-04-24T14:35:22.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001227 2024-04-24T14:35:22.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:22.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.010194+0000", 2024-04-24T14:35:22.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040549380000000001 2024-04-24T14:35:22.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:22.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020365+0000", 2024-04-24T14:35:22.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010170983 2024-04-24T14:35:22.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:22.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020428+0000", 2024-04-24T14:35:22.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2457000000000002e-05 2024-04-24T14:35:22.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020492+0000", 2024-04-24T14:35:22.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4592999999999996e-05 2024-04-24T14:35:22.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.259 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.259 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:155 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1601536~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:22.260 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:22.260 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.129857926, 2024-04-24T14:35:22.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015144391, 2024-04-24T14:35:22.260 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.260 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:22.260 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.260 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:22.260 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:22.260 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 155 2024-04-24T14:35:22.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.260 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:22.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039507+0000", 2024-04-24T14:35:22.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039506+0000", 2024-04-24T14:35:22.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:22.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039766+0000", 2024-04-24T14:35:22.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025994400000000002 2024-04-24T14:35:22.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039780+0000", 2024-04-24T14:35:22.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3366e-05 2024-04-24T14:35:22.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039787+0000", 2024-04-24T14:35:22.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4359999999999996e-06 2024-04-24T14:35:22.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.050886+0000", 2024-04-24T14:35:22.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011098993 2024-04-24T14:35:22.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051026+0000", 2024-04-24T14:35:22.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001395 2024-04-24T14:35:22.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:22.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051211+0000", 2024-04-24T14:35:22.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018547099999999999 2024-04-24T14:35:22.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:22.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054490+0000", 2024-04-24T14:35:22.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032785499999999999 2024-04-24T14:35:22.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:22.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054505+0000", 2024-04-24T14:35:22.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5906999999999999e-05 2024-04-24T14:35:22.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:22.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054594+0000", 2024-04-24T14:35:22.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8817e-05 2024-04-24T14:35:22.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054651+0000", 2024-04-24T14:35:22.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7040999999999998e-05 2024-04-24T14:35:22.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.265 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.266 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:156 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1732608~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:22.266 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:22.266 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.129567219999998, 2024-04-24T14:35:22.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016895194999999998, 2024-04-24T14:35:22.266 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.266 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:22.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:22.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:22.266 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 156 2024-04-24T14:35:22.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039798+0000", 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.039797+0000", 2024-04-24T14:35:22.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:22.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040065+0000", 2024-04-24T14:35:22.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000268361 2024-04-24T14:35:22.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040086+0000", 2024-04-24T14:35:22.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1024000000000001e-05 2024-04-24T14:35:22.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040094+0000", 2024-04-24T14:35:22.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9379999999999992e-06 2024-04-24T14:35:22.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.051936+0000", 2024-04-24T14:35:22.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011842518 2024-04-24T14:35:22.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052062+0000", 2024-04-24T14:35:22.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000125482 2024-04-24T14:35:22.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:22.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.052206+0000", 2024-04-24T14:35:22.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143953 2024-04-24T14:35:22.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:22.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054715+0000", 2024-04-24T14:35:22.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025088240000000002 2024-04-24T14:35:22.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:22.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056534+0000", 2024-04-24T14:35:22.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001819436 2024-04-24T14:35:22.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:22.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056610+0000", 2024-04-24T14:35:22.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5383000000000007e-05 2024-04-24T14:35:22.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056693+0000", 2024-04-24T14:35:22.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3283999999999997e-05 2024-04-24T14:35:22.271 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.271 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.272 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.272 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:157 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1863680~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:22.272 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:22.272 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.129259841, 2024-04-24T14:35:22.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016763674999999999, 2024-04-24T14:35:22.272 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.272 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:22.272 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.272 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:22.272 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:22.272 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 157 2024-04-24T14:35:22.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.273 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:22.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040105+0000", 2024-04-24T14:35:22.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040104+0000", 2024-04-24T14:35:22.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:22.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040244+0000", 2024-04-24T14:35:22.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014035499999999999 2024-04-24T14:35:22.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040255+0000", 2024-04-24T14:35:22.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0135999999999999e-05 2024-04-24T14:35:22.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040260+0000", 2024-04-24T14:35:22.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7919999999999998e-06 2024-04-24T14:35:22.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053406+0000", 2024-04-24T14:35:22.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013146023999999999 2024-04-24T14:35:22.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053531+0000", 2024-04-24T14:35:22.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012471800000000001 2024-04-24T14:35:22.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:22.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.053715+0000", 2024-04-24T14:35:22.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018395099999999999 2024-04-24T14:35:22.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:22.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056711+0000", 2024-04-24T14:35:22.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029956520000000001 2024-04-24T14:35:22.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:22.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056783+0000", 2024-04-24T14:35:22.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2010999999999999e-05 2024-04-24T14:35:22.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:22.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056831+0000", 2024-04-24T14:35:22.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8189999999999998e-05 2024-04-24T14:35:22.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056869+0000", 2024-04-24T14:35:22.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7812000000000003e-05 2024-04-24T14:35:22.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.278 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.278 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:158 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1994752~131072] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:22.278 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:22.278 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.129097913999999, 2024-04-24T14:35:22.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016845334999999999, 2024-04-24T14:35:22.278 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.278 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:22.278 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.278 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040267+0000", 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040266+0000", 2024-04-24T14:35:22.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:22.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040379+0000", 2024-04-24T14:35:22.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112728 2024-04-24T14:35:22.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040387+0000", 2024-04-24T14:35:22.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8499999999999994e-06 2024-04-24T14:35:22.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.040391+0000", 2024-04-24T14:35:22.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2719999999999999e-06 2024-04-24T14:35:22.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054803+0000", 2024-04-24T14:35:22.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014411548 2024-04-24T14:35:22.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.054982+0000", 2024-04-24T14:35:22.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017941699999999999 2024-04-24T14:35:22.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4", 2024-04-24T14:35:22.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.055167+0000", 2024-04-24T14:35:22.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018542100000000001 2024-04-24T14:35:22.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:22.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056718+0000", 2024-04-24T14:35:22.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015505670000000001 2024-04-24T14:35:22.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057020+0000", 2024-04-24T14:35:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030227500000000002 2024-04-24T14:35:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057070+0000", 2024-04-24T14:35:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9611000000000003e-05 2024-04-24T14:35:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057112+0000", 2024-04-24T14:35:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2373999999999998e-05 2024-04-24T14:35:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:159 3.1 3:82cd38f2:::rbd_data.12ff64619e2d.0000000000000000:head [read 28672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.123662926000002, 2024-04-24T14:35:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015930650000000001, 2024-04-24T14:35:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045702+0000", 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045701+0000", 2024-04-24T14:35:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045707+0000", 2024-04-24T14:35:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.562e-06 2024-04-24T14:35:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045728+0000", 2024-04-24T14:35:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1109999999999999e-05 2024-04-24T14:35:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.045738+0000", 2024-04-24T14:35:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0016e-05 2024-04-24T14:35:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056183+0000", 2024-04-24T14:35:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010444836000000001 2024-04-24T14:35:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.056470+0000", 2024-04-24T14:35:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028696899999999998 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057160+0000", 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069004699999999995 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.057291+0000", 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013096200000000001 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.061633+0000", 2024-04-24T14:35:22.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.004341414 2024-04-24T14:35:22.289 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.289 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.289 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.289 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.289 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.289 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:22.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-24T14:35:22.396 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:22.396 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:22.396 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:35:22.396 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:22.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-24T14:35:22.428 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2024-04-24T14:35:22.504 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:22.504 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:22.504 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:22.505 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:22.505 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:22.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-24T14:35:22.617 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:22.617 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:22.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:22.617 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:22.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.617 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:193 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:22.617 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:22.618 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 101.14981139699999, 2024-04-24T14:35:22.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011316720000000001, 2024-04-24T14:35:22.618 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.618 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.618 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2024-04-24T14:35:22.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.618 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:22.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471123+0000", 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471122+0000", 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471126+0000", 2024-04-24T14:35:22.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.76e-06 2024-04-24T14:35:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471134+0000", 2024-04-24T14:35:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9929999999999995e-06 2024-04-24T14:35:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.471139+0000", 2024-04-24T14:35:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.186e-06 2024-04-24T14:35:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481261+0000", 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010121889 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.481512+0000", 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025059799999999999 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.482440+0000", 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00092809999999999995 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:525 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3407872~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.774454694, 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010866061999999999, 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846480+0000", 2024-04-24T14:35:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846479+0000", 2024-04-24T14:35:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846484+0000", 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6540000000000004e-06 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846493+0000", 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3469999999999999e-06 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.846498+0000", 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856332+0000", 2024-04-24T14:35:22.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098341750000000006 2024-04-24T14:35:22.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.856552+0000", 2024-04-24T14:35:22.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021964 2024-04-24T14:35:22.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.857346+0000", 2024-04-24T14:35:22.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00079389300000000005 2024-04-24T14:35:22.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.626 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.626 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:527 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3801088~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.626 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:22.626 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.77302654, 2024-04-24T14:35:22.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011369433, 2024-04-24T14:35:22.626 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.626 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.626 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2024-04-24T14:35:22.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.627 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:22.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847908+0000", 2024-04-24T14:35:22.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847907+0000", 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847911+0000", 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4850000000000003e-06 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847922+0000", 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0652000000000001e-05 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847928+0000", 2024-04-24T14:35:22.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999999e-06 2024-04-24T14:35:22.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858478+0000", 2024-04-24T14:35:22.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01055042 2024-04-24T14:35:22.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.858564+0000", 2024-04-24T14:35:22.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5884999999999995e-05 2024-04-24T14:35:22.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859277+0000", 2024-04-24T14:35:22.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071317400000000001 2024-04-24T14:35:22.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.630 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.630 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:528 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.630 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:22.630 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.772999451, 2024-04-24T14:35:22.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012392884999999999, 2024-04-24T14:35:22.631 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.631 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.631 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 528 2024-04-24T14:35:22.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.631 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:22.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847935+0000", 2024-04-24T14:35:22.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847934+0000", 2024-04-24T14:35:22.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:22.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847938+0000", 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6629999999999999e-06 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847947+0000", 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5219999999999996e-06 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847953+0000", 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2580000000000003e-06 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859329+0000", 2024-04-24T14:35:22.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011376571 2024-04-24T14:35:22.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.859558+0000", 2024-04-24T14:35:22.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022902900000000001 2024-04-24T14:35:22.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860328+0000", 2024-04-24T14:35:22.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076947199999999999 2024-04-24T14:35:22.634 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.634 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.634 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.635 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:529 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.635 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:22.635 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.772975484, 2024-04-24T14:35:22.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013745724000000001, 2024-04-24T14:35:22.635 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.635 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.635 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.635 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.635 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.635 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 529 2024-04-24T14:35:22.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.635 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847959+0000", 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847958+0000", 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:22.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847962+0000", 2024-04-24T14:35:22.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.664e-06 2024-04-24T14:35:22.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847984+0000", 2024-04-24T14:35:22.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1770999999999999e-05 2024-04-24T14:35:22.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.847989+0000", 2024-04-24T14:35:22.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6500000000000001e-06 2024-04-24T14:35:22.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860432+0000", 2024-04-24T14:35:22.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012442656999999999 2024-04-24T14:35:22.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.860765+0000", 2024-04-24T14:35:22.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000333223 2024-04-24T14:35:22.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861705+0000", 2024-04-24T14:35:22.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093939000000000002 2024-04-24T14:35:22.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.639 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.639 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:531 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.639 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:22.639 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.758991505, 2024-04-24T14:35:22.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.028391280000000001, 2024-04-24T14:35:22.639 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.639 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.640 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.640 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 531 2024-04-24T14:35:22.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.640 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:22.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861943+0000", 2024-04-24T14:35:22.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861942+0000", 2024-04-24T14:35:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861949+0000", 2024-04-24T14:35:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9979999999999997e-06 2024-04-24T14:35:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861966+0000", 2024-04-24T14:35:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7312e-05 2024-04-24T14:35:22.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.861976+0000", 2024-04-24T14:35:22.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0512e-05 2024-04-24T14:35:22.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862529+0000", 2024-04-24T14:35:22.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055287999999999995 2024-04-24T14:35:22.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862659+0000", 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008799999999999 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885085+0000", 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022425558000000002 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885217+0000", 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013206000000000001 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.888927+0000", 2024-04-24T14:35:22.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037099450000000001 2024-04-24T14:35:22.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.889002+0000", 2024-04-24T14:35:22.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5459999999999999e-05 2024-04-24T14:35:22.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890334+0000", 2024-04-24T14:35:22.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001331689 2024-04-24T14:35:22.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.644 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:532 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 262144~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.758875194, 2024-04-24T14:35:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029010706000000001, 2024-04-24T14:35:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 532 2024-04-24T14:35:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:22.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862059+0000", 2024-04-24T14:35:22.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862057+0000", 2024-04-24T14:35:22.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:22.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862066+0000", 2024-04-24T14:35:22.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3829999999999998e-06 2024-04-24T14:35:22.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862083+0000", 2024-04-24T14:35:22.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6881999999999999e-05 2024-04-24T14:35:22.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862095+0000", 2024-04-24T14:35:22.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1952e-05 2024-04-24T14:35:22.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862744+0000", 2024-04-24T14:35:22.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064841699999999998 2024-04-24T14:35:22.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862820+0000", 2024-04-24T14:35:22.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6763999999999999e-05 2024-04-24T14:35:22.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885274+0000", 2024-04-24T14:35:22.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.022453956000000001 2024-04-24T14:35:22.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885362+0000", 2024-04-24T14:35:22.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7260000000000001e-05 2024-04-24T14:35:22.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890368+0000", 2024-04-24T14:35:22.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050062170000000003 2024-04-24T14:35:22.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.890454+0000", 2024-04-24T14:35:22.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6526999999999993e-05 2024-04-24T14:35:22.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891070+0000", 2024-04-24T14:35:22.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000615462 2024-04-24T14:35:22.651 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.651 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.651 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.651 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:533 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 393216~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.651 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:22.651 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.758827536, 2024-04-24T14:35:22.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029683943000000001, 2024-04-24T14:35:22.651 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.651 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.651 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.651 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.651 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.652 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 533 2024-04-24T14:35:22.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.652 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:22.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862107+0000", 2024-04-24T14:35:22.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862106+0000", 2024-04-24T14:35:22.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:22.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862111+0000", 2024-04-24T14:35:22.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6400000000000002e-06 2024-04-24T14:35:22.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862124+0000", 2024-04-24T14:35:22.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2972e-05 2024-04-24T14:35:22.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862133+0000", 2024-04-24T14:35:22.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5199999999999997e-06 2024-04-24T14:35:22.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.862888+0000", 2024-04-24T14:35:22.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075549500000000002 2024-04-24T14:35:22.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.863001+0000", 2024-04-24T14:35:22.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011212199999999999 2024-04-24T14:35:22.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885442+0000", 2024-04-24T14:35:22.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02244165 2024-04-24T14:35:22.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885583+0000", 2024-04-24T14:35:22.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014086799999999999 2024-04-24T14:35:22.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891101+0000", 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055183740000000004 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891192+0000", 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0835999999999996e-05 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891791+0000", 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059841600000000005 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:534 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 524288~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.754790238, 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026345324999999999, 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 534 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866144+0000", 2024-04-24T14:35:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866143+0000", 2024-04-24T14:35:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866151+0000", 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3650000000000004e-06 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866172+0000", 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0735000000000001e-05 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866182+0000", 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0688e-05 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3566000000000002e-05 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866268+0000", 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2036000000000003e-05 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885639+0000", 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019370973999999999 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885707+0000", 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8036999999999999e-05 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891821+0000", 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0061141010000000003 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.891899+0000", 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7631000000000003e-05 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.892489+0000", 2024-04-24T14:35:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059075200000000005 2024-04-24T14:35:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:535 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 786432~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.754726479, 2024-04-24T14:35:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027732974000000001, 2024-04-24T14:35:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 535 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866208+0000", 2024-04-24T14:35:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866207+0000", 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866211+0000", 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866219+0000", 2024-04-24T14:35:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1669999999999999e-06 2024-04-24T14:35:22.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866227+0000", 2024-04-24T14:35:22.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8599999999999993e-06 2024-04-24T14:35:22.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866317+0000", 2024-04-24T14:35:22.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0457000000000005e-05 2024-04-24T14:35:22.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866373+0000", 2024-04-24T14:35:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5381999999999999e-05 2024-04-24T14:35:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885763+0000", 2024-04-24T14:35:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019390678000000001 2024-04-24T14:35:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885893+0000", 2024-04-24T14:35:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129176 2024-04-24T14:35:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893278+0000", 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0073853770000000003 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893340+0000", 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1711999999999995e-05 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.893941+0000", 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000601064 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:536 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 655360~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.754700989, 2024-04-24T14:35:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.02849235, 2024-04-24T14:35:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 536 2024-04-24T14:35:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:22.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:22.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866233+0000", 2024-04-24T14:35:22.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866236+0000", 2024-04-24T14:35:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2100000000000002e-06 2024-04-24T14:35:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866246+0000", 2024-04-24T14:35:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.047e-05 2024-04-24T14:35:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866251+0000", 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8230000000000002e-06 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866405+0000", 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015411900000000001 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866425+0000", 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9463999999999999e-05 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.885939+0000", 2024-04-24T14:35:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019514179 2024-04-24T14:35:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886008+0000", 2024-04-24T14:35:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9422e-05 2024-04-24T14:35:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894004+0000", 2024-04-24T14:35:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079957129999999998 2024-04-24T14:35:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894116+0000", 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112163 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894726+0000", 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060942400000000001 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:537 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 917504~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.754677239, 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029265332000000002, 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 537 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866257+0000", 2024-04-24T14:35:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866256+0000", 2024-04-24T14:35:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866260+0000", 2024-04-24T14:35:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5360000000000001e-06 2024-04-24T14:35:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866267+0000", 2024-04-24T14:35:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8850000000000001e-06 2024-04-24T14:35:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866271+0000", 2024-04-24T14:35:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5260000000000004e-06 2024-04-24T14:35:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866460+0000", 2024-04-24T14:35:22.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018917700000000001 2024-04-24T14:35:22.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866510+0000", 2024-04-24T14:35:22.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9184000000000002e-05 2024-04-24T14:35:22.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886065+0000", 2024-04-24T14:35:22.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019554919 2024-04-24T14:35:22.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886192+0000", 2024-04-24T14:35:22.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012788200000000001 2024-04-24T14:35:22.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894796+0000", 2024-04-24T14:35:22.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008603177 2024-04-24T14:35:22.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.894877+0000", 2024-04-24T14:35:22.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1081999999999994e-05 2024-04-24T14:35:22.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895522+0000", 2024-04-24T14:35:22.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064570899999999995 2024-04-24T14:35:22.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.679 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.679 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:538 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1048576~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.679 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:22.680 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.754638726, 2024-04-24T14:35:22.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030071365999999999, 2024-04-24T14:35:22.680 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.680 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.680 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.680 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.680 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.680 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 538 2024-04-24T14:35:22.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.680 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:22.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866296+0000", 2024-04-24T14:35:22.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866295+0000", 2024-04-24T14:35:22.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:22.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866305+0000", 2024-04-24T14:35:22.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0254e-05 2024-04-24T14:35:22.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866313+0000", 2024-04-24T14:35:22.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9000000000000006e-06 2024-04-24T14:35:22.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866318+0000", 2024-04-24T14:35:22.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3530000000000002e-06 2024-04-24T14:35:22.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866543+0000", 2024-04-24T14:35:22.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022500999999999999 2024-04-24T14:35:22.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866567+0000", 2024-04-24T14:35:22.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3594000000000001e-05 2024-04-24T14:35:22.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886244+0000", 2024-04-24T14:35:22.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019677315000000001 2024-04-24T14:35:22.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886312+0000", 2024-04-24T14:35:22.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7414000000000002e-05 2024-04-24T14:35:22.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895565+0000", 2024-04-24T14:35:22.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092527830000000005 2024-04-24T14:35:22.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.895662+0000", 2024-04-24T14:35:22.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7025e-05 2024-04-24T14:35:22.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896367+0000", 2024-04-24T14:35:22.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00070540299999999995 2024-04-24T14:35:22.685 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.685 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.685 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.685 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:539 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1179648~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.685 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:22.685 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.754610529, 2024-04-24T14:35:22.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030884712000000002, 2024-04-24T14:35:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 539 2024-04-24T14:35:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866324+0000", 2024-04-24T14:35:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866323+0000", 2024-04-24T14:35:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866326+0000", 2024-04-24T14:35:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1829999999999998e-06 2024-04-24T14:35:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866333+0000", 2024-04-24T14:35:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.353e-06 2024-04-24T14:35:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866337+0000", 2024-04-24T14:35:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3150000000000002e-06 2024-04-24T14:35:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866596+0000", 2024-04-24T14:35:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000258769 2024-04-24T14:35:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866634+0000", 2024-04-24T14:35:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8260999999999998e-05 2024-04-24T14:35:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886366+0000", 2024-04-24T14:35:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019731504 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886490+0000", 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124695 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896448+0000", 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0099580350000000005 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.896567+0000", 2024-04-24T14:35:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118861 2024-04-24T14:35:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897209+0000", 2024-04-24T14:35:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064135699999999995 2024-04-24T14:35:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:540 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1310720~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.754589072, 2024-04-24T14:35:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.031711479000000001, 2024-04-24T14:35:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 540 2024-04-24T14:35:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866345+0000", 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866344+0000", 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866348+0000", 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3330000000000001e-06 2024-04-24T14:35:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866354+0000", 2024-04-24T14:35:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2024-04-24T14:35:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866359+0000", 2024-04-24T14:35:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:35:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866664+0000", 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000305679 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866693+0000", 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8357999999999999e-05 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886540+0000", 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019847329 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886601+0000", 2024-04-24T14:35:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0783999999999999e-05 2024-04-24T14:35:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897264+0000", 2024-04-24T14:35:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010662939 2024-04-24T14:35:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.897379+0000", 2024-04-24T14:35:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114873 2024-04-24T14:35:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898057+0000", 2024-04-24T14:35:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067816299999999996 2024-04-24T14:35:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5007.0:541 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54)", 2024-04-24T14:35:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.75457023, 2024-04-24T14:35:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032500854000000003, 2024-04-24T14:35:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5007", 2024-04-24T14:35:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 541 2024-04-24T14:35:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866364+0000", 2024-04-24T14:35:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866363+0000", 2024-04-24T14:35:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866368+0000", 2024-04-24T14:35:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1479999999999999e-06 2024-04-24T14:35:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866374+0000", 2024-04-24T14:35:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5370000000000003e-06 2024-04-24T14:35:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866380+0000", 2024-04-24T14:35:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0800000000000002e-06 2024-04-24T14:35:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866711+0000", 2024-04-24T14:35:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033118500000000001 2024-04-24T14:35:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.866739+0000", 2024-04-24T14:35:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7182000000000002e-05 2024-04-24T14:35:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886638+0000", 2024-04-24T14:35:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01989956 2024-04-24T14:35:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.886761+0000", 2024-04-24T14:35:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012292999999999999 2024-04-24T14:35:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898133+0000", 2024-04-24T14:35:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011371902 2024-04-24T14:35:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898234+0000", 2024-04-24T14:35:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100567 2024-04-24T14:35:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:41.898865+0000", 2024-04-24T14:35:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063137299999999996 2024-04-24T14:35:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5048.0:42 3.0 3:2a56efe6:ns::rbd_data.1294a8af6a1d.000000000000000f:5 [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 83.541738081000005, 2024-04-24T14:35:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025764797999999998, 2024-04-24T14:35:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5048", 2024-04-24T14:35:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 42 2024-04-24T14:35:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079196+0000", 2024-04-24T14:35:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079195+0000", 2024-04-24T14:35:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079205+0000", 2024-04-24T14:35:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2024-04-24T14:35:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079232+0000", 2024-04-24T14:35:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:35:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079243+0000", 2024-04-24T14:35:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.151e-05 2024-04-24T14:35:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079397+0000", 2024-04-24T14:35:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015312600000000001 2024-04-24T14:35:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.079474+0000", 2024-04-24T14:35:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7008999999999994e-05 2024-04-24T14:35:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.098997+0000", 2024-04-24T14:35:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.019523248999999999 2024-04-24T14:35:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.099099+0000", 2024-04-24T14:35:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010231800000000001 2024-04-24T14:35:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101928+0000", 2024-04-24T14:35:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0028291660000000001 2024-04-24T14:35:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.101989+0000", 2024-04-24T14:35:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0423000000000001e-05 2024-04-24T14:35:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:33:59.104961+0000", 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002972354 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:39 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1753088~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.601965665000002, 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01139394, 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 39 2024-04-24T14:35:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018969+0000", 2024-04-24T14:35:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.018967+0000", 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019242+0000", 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027504500000000002 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019274+0000", 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2472999999999999e-05 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019288+0000", 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3283e-05 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026753+0000", 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074651120000000003 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.026971+0000", 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021810800000000001 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.027211+0000", 2024-04-24T14:35:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023975 2024-04-24T14:35:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030200+0000", 2024-04-24T14:35:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0029890709999999998 2024-04-24T14:35:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030252+0000", 2024-04-24T14:35:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1926999999999999e-05 2024-04-24T14:35:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030312+0000", 2024-04-24T14:35:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0602e-05 2024-04-24T14:35:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030363+0000", 2024-04-24T14:35:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0342e-05 2024-04-24T14:35:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:40 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 1884160~131072] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.601629690999999, 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011199064, 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 40 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019305+0000", 2024-04-24T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019303+0000", 2024-04-24T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019729+0000", 2024-04-24T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042620899999999998 2024-04-24T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019769+0000", 2024-04-24T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9847e-05 2024-04-24T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019797+0000", 2024-04-24T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8347000000000002e-05 2024-04-24T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028299+0000", 2024-04-24T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0085010629999999997 2024-04-24T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028385+0000", 2024-04-24T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6824999999999996e-05 2024-04-24T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.028510+0000", 2024-04-24T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012483000000000001 2024-04-24T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030410+0000", 2024-04-24T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001899612 2024-04-24T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030416+0000", 2024-04-24T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7699999999999998e-06 2024-04-24T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030461+0000", 2024-04-24T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5157999999999997e-05 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030504+0000", 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2988999999999998e-05 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5769.0:41 3.0 3:3ad083fe:::rbd_data.12ff64619e2d.0000000000000001:head [write 2015232~81920] snapc 8=[8,7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.601120983999998, 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012345257, 2024-04-24T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5769", 2024-04-24T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1460942673", 2024-04-24T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 41 2024-04-24T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019813+0000", 2024-04-24T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019812+0000", 2024-04-24T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.019975+0000", 2024-04-24T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016325700000000001 2024-04-24T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020025+0000", 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9484999999999998e-05 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.020035+0000", 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0444e-05 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029150+0000", 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0091150160000000001 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029352+0000", 2024-04-24T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020166499999999999 2024-04-24T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1", 2024-04-24T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.029611+0000", 2024-04-24T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025949600000000001 2024-04-24T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.030518+0000", 2024-04-24T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090633300000000005 2024-04-24T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032088+0000", 2024-04-24T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015703290000000001 2024-04-24T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032130+0000", 2024-04-24T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2273e-05 2024-04-24T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:31.032159+0000", 2024-04-24T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8387000000000001e-05 2024-04-24T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:22.727 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-24T14:35:22.836 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:22.836 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:22.836 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:35:22.836 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:22.845 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-24T14:35:22.948 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:35:22.948 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:35:22.948 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:22.957 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-24T14:35:23.058 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:23.058 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:35:23.058 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:35:23.058 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:35:23.058 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.059 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:23.059 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:23.059 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 180.43474064700001, 2024-04-24T14:35:23.059 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015522764, 2024-04-24T14:35:23.059 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.059 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.059 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.059 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:23.059 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:23.059 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 110 2024-04-24T14:35:23.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.059 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:23.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.060 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.060 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625803+0000", 2024-04-24T14:35:23.060 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.060 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625802+0000", 2024-04-24T14:35:23.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:23.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625814+0000", 2024-04-24T14:35:23.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2373e-05 2024-04-24T14:35:23.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.061 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.061 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.061 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625832+0000", 2024-04-24T14:35:23.061 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7762999999999999e-05 2024-04-24T14:35:23.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625842+0000", 2024-04-24T14:35:23.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0200000000000001e-05 2024-04-24T14:35:23.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633380+0000", 2024-04-24T14:35:23.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0075379699999999997 2024-04-24T14:35:23.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.062 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.062 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.062 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633459+0000", 2024-04-24T14:35:23.062 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9289999999999995e-05 2024-04-24T14:35:23.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:23.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.633588+0000", 2024-04-24T14:35:23.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012848699999999999 2024-04-24T14:35:23.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.063 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.063 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:23.063 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636684+0000", 2024-04-24T14:35:23.063 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030957459999999999 2024-04-24T14:35:23.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:23.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641218+0000", 2024-04-24T14:35:23.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0045346850000000001 2024-04-24T14:35:23.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641285+0000", 2024-04-24T14:35:23.064 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6440999999999996e-05 2024-04-24T14:35:23.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.064 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.064 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.064 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641326+0000", 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1214999999999998e-05 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 180.43456355999999, 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015519775, 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:23.065 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:23.066 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 111 2024-04-24T14:35:23.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.066 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:23.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.066 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.066 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.066 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625980+0000", 2024-04-24T14:35:23.066 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625979+0000", 2024-04-24T14:35:23.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:23.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.625993+0000", 2024-04-24T14:35:23.067 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4019000000000001e-05 2024-04-24T14:35:23.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.067 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.067 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.067 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626041+0000", 2024-04-24T14:35:23.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7882e-05 2024-04-24T14:35:23.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626056+0000", 2024-04-24T14:35:23.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5157000000000001e-05 2024-04-24T14:35:23.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.068 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634082+0000", 2024-04-24T14:35:23.068 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080258169999999993 2024-04-24T14:35:23.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.068 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.068 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634164+0000", 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1924999999999999e-05 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.634302+0000", 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013874300000000001 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.640908+0000", 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0066056359999999998 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.069 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:23.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641343+0000", 2024-04-24T14:35:23.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00043477499999999998 2024-04-24T14:35:23.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641438+0000", 2024-04-24T14:35:23.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4802000000000005e-05 2024-04-24T14:35:23.070 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.070 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.070 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.070 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641500+0000", 2024-04-24T14:35:23.070 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2638999999999994e-05 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 180.43445964599999, 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015601594999999999, 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:23.071 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 112 2024-04-24T14:35:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626084+0000", 2024-04-24T14:35:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.072 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626082+0000", 2024-04-24T14:35:23.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:23.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626293+0000", 2024-04-24T14:35:23.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021095200000000001 2024-04-24T14:35:23.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.073 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.073 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.073 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626332+0000", 2024-04-24T14:35:23.073 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8859000000000002e-05 2024-04-24T14:35:23.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626358+0000", 2024-04-24T14:35:23.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6789000000000001e-05 2024-04-24T14:35:23.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635232+0000", 2024-04-24T14:35:23.074 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0088736349999999995 2024-04-24T14:35:23.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.074 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.074 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.074 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635351+0000", 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000119246 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.635484+0000", 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013224999999999999 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641095+0000", 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0056111019999999998 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641536+0000", 2024-04-24T14:35:23.075 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00044125000000000002 2024-04-24T14:35:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641640+0000", 2024-04-24T14:35:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103812 2024-04-24T14:35:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641686+0000", 2024-04-24T14:35:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6156999999999999e-05 2024-04-24T14:35:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.076 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 180.434071195, 2024-04-24T14:35:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015642001999999999, 2024-04-24T14:35:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:23.077 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 114 2024-04-24T14:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626473+0000", 2024-04-24T14:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626471+0000", 2024-04-24T14:35:23.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:23.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626483+0000", 2024-04-24T14:35:23.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1846e-05 2024-04-24T14:35:23.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.079 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.079 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.079 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626513+0000", 2024-04-24T14:35:23.079 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9899e-05 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626539+0000", 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6322999999999999e-05 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636742+0000", 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010202767999999999 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636840+0000", 2024-04-24T14:35:23.080 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7430999999999996e-05 2024-04-24T14:35:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.636949+0000", 2024-04-24T14:35:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109663 2024-04-24T14:35:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641907+0000", 2024-04-24T14:35:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049578469999999996 2024-04-24T14:35:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:23.081 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641997+0000", 2024-04-24T14:35:23.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0049999999999993e-05 2024-04-24T14:35:23.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642061+0000", 2024-04-24T14:35:23.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3919999999999998e-05 2024-04-24T14:35:23.082 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.082 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.082 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.082 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642115+0000", 2024-04-24T14:35:23.082 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3730999999999998e-05 2024-04-24T14:35:23.082 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.082 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.082 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 180.43398598300001, 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017600602999999999, 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626558+0000", 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626557+0000", 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.084 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626738+0000", 2024-04-24T14:35:23.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018119300000000001 2024-04-24T14:35:23.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626761+0000", 2024-04-24T14:35:23.085 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3079e-05 2024-04-24T14:35:23.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.085 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.085 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.085 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626776+0000", 2024-04-24T14:35:23.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4919999999999999e-05 2024-04-24T14:35:23.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637595+0000", 2024-04-24T14:35:23.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010818716000000001 2024-04-24T14:35:23.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.086 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.086 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637673+0000", 2024-04-24T14:35:23.086 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8786000000000001e-05 2024-04-24T14:35:23.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.086 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:23.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.637805+0000", 2024-04-24T14:35:23.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013172499999999999 2024-04-24T14:35:23.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:23.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.641976+0000", 2024-04-24T14:35:23.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041708090000000001 2024-04-24T14:35:23.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.087 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.087 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:23.087 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643978+0000", 2024-04-24T14:35:23.087 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0020019450000000002 2024-04-24T14:35:23.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644081+0000", 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103169 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.644159+0000", 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7478000000000006e-05 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:23.088 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:23.089 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 180.43374910200001, 2024-04-24T14:35:23.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019973798000000001, 2024-04-24T14:35:23.089 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.089 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.089 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.089 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:23.089 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:23.089 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 116 2024-04-24T14:35:23.089 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.089 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.089 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.089 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.089 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:23.089 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626795+0000", 2024-04-24T14:35:23.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.090 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.090 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626793+0000", 2024-04-24T14:35:23.090 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:23.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.090 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626805+0000", 2024-04-24T14:35:23.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2169e-05 2024-04-24T14:35:23.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.091 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626842+0000", 2024-04-24T14:35:23.091 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6619000000000003e-05 2024-04-24T14:35:23.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.091 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.091 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626860+0000", 2024-04-24T14:35:23.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7873e-05 2024-04-24T14:35:23.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642151+0000", 2024-04-24T14:35:23.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015290859 2024-04-24T14:35:23.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.092 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.092 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642259+0000", 2024-04-24T14:35:23.092 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000108553 2024-04-24T14:35:23.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.092 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:23.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642425+0000", 2024-04-24T14:35:23.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000166136 2024-04-24T14:35:23.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:23.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645513+0000", 2024-04-24T14:35:23.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0030875640000000001 2024-04-24T14:35:23.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.093 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:23.093 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646606+0000", 2024-04-24T14:35:23.093 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001093526 2024-04-24T14:35:23.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646693+0000", 2024-04-24T14:35:23.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6929999999999996e-05 2024-04-24T14:35:23.094 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.094 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.094 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.094 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646769+0000", 2024-04-24T14:35:23.094 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5375000000000003e-05 2024-04-24T14:35:23.094 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.094 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.095 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.095 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.095 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:23.095 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:23.095 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 180.43366757999999, 2024-04-24T14:35:23.095 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020113991000000001, 2024-04-24T14:35:23.095 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.095 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.095 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.095 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:23.095 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:23.095 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 117 2024-04-24T14:35:23.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.096 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:23.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.096 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.096 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.096 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626876+0000", 2024-04-24T14:35:23.096 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.096 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626875+0000", 2024-04-24T14:35:23.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:23.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626935+0000", 2024-04-24T14:35:23.097 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9759000000000002e-05 2024-04-24T14:35:23.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.097 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.097 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.097 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626957+0000", 2024-04-24T14:35:23.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2089000000000001e-05 2024-04-24T14:35:23.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.626968+0000", 2024-04-24T14:35:23.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0886999999999999e-05 2024-04-24T14:35:23.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.098 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.098 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642820+0000", 2024-04-24T14:35:23.098 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015852763999999998 2024-04-24T14:35:23.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.642929+0000", 2024-04-24T14:35:23.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010809799999999999 2024-04-24T14:35:23.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:23.099 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.643089+0000", 2024-04-24T14:35:23.099 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016037000000000001 2024-04-24T14:35:23.099 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.099 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.099 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:23.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.645584+0000", 2024-04-24T14:35:23.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024954249999999999 2024-04-24T14:35:23.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:23.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646790+0000", 2024-04-24T14:35:23.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0012053179999999999 2024-04-24T14:35:23.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.100 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.100 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.100 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646907+0000", 2024-04-24T14:35:23.100 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117491 2024-04-24T14:35:23.100 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:22.646990+0000", 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3164000000000005e-05 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 180.00175523300001, 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016379312, 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.101 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.102 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.102 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:23.102 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:23.102 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 138 2024-04-24T14:35:23.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.102 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.102 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.102 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:23.102 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.102 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.102 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058789+0000", 2024-04-24T14:35:23.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058788+0000", 2024-04-24T14:35:23.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:23.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.103 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.103 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.103 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058794+0000", 2024-04-24T14:35:23.103 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8150000000000002e-06 2024-04-24T14:35:23.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058800+0000", 2024-04-24T14:35:23.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6810000000000001e-06 2024-04-24T14:35:23.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058807+0000", 2024-04-24T14:35:23.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2199999999999997e-06 2024-04-24T14:35:23.104 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.104 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.104 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.104 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068754+0000", 2024-04-24T14:35:23.104 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0099478299999999995 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.068862+0000", 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107475 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.069043+0000", 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018141000000000001 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.074912+0000", 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058681899999999997 2024-04-24T14:35:23.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:23.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075069+0000", 2024-04-24T14:35:23.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000157236 2024-04-24T14:35:23.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.106 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.106 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.106 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075129+0000", 2024-04-24T14:35:23.106 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0177999999999999e-05 2024-04-24T14:35:23.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.107 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.107 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075168+0000", 2024-04-24T14:35:23.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9076999999999999e-05 2024-04-24T14:35:23.107 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.107 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.107 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.107 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.107 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:23.107 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:23.107 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 180.00172693299999, 2024-04-24T14:35:23.107 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016436256999999999, 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 139 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.108 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058817+0000", 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058816+0000", 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058822+0000", 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0919999999999997e-06 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.109 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058830+0000", 2024-04-24T14:35:23.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7830000000000004e-06 2024-04-24T14:35:23.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058834+0000", 2024-04-24T14:35:23.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.535e-06 2024-04-24T14:35:23.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.110 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.110 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070248+0000", 2024-04-24T14:35:23.110 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011414218 2024-04-24T14:35:23.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070367+0000", 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000118974 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.070531+0000", 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000164098 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075013+0000", 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044819250000000003 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075184+0000", 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00017081099999999999 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075224+0000", 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0515000000000001e-05 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075253+0000", 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.8892000000000001e-05 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.112 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 180.00163922900001, 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.017002592, 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 140 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.113 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:23.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.114 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.114 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058905+0000", 2024-04-24T14:35:23.114 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.114 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.114 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.058904+0000", 2024-04-24T14:35:23.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:23.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059003+0000", 2024-04-24T14:35:23.115 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.9301000000000004e-05 2024-04-24T14:35:23.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.115 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.115 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.115 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059013+0000", 2024-04-24T14:35:23.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7459999999999995e-06 2024-04-24T14:35:23.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059019+0000", 2024-04-24T14:35:23.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7010000000000001e-06 2024-04-24T14:35:23.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.116 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.116 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.116 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071070+0000", 2024-04-24T14:35:23.116 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012051512 2024-04-24T14:35:23.116 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071164+0000", 2024-04-24T14:35:23.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.4260999999999998e-05 2024-04-24T14:35:23.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:23.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.071312+0000", 2024-04-24T14:35:23.117 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014732800000000001 2024-04-24T14:35:23.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.117 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.117 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:23.117 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075055+0000", 2024-04-24T14:35:23.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037432419999999999 2024-04-24T14:35:23.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:23.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075788+0000", 2024-04-24T14:35:23.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00073286100000000002 2024-04-24T14:35:23.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.118 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075856+0000", 2024-04-24T14:35:23.118 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8516000000000006e-05 2024-04-24T14:35:23.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.118 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.118 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.119 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075907+0000", 2024-04-24T14:35:23.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0918999999999998e-05 2024-04-24T14:35:23.119 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.119 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.119 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.119 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.119 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4192.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-04-24T14:35:23.119 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:23.119 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 180.001516428, 2024-04-24T14:35:23.119 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.016972391999999999, 2024-04-24T14:35:23.119 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.119 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.119 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.120 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4192", 2024-04-24T14:35:23.120 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.103:0/764388718", 2024-04-24T14:35:23.120 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 141 2024-04-24T14:35:23.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.120 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:23.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.120 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.120 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.120 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059028+0000", 2024-04-24T14:35:23.120 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059027+0000", 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059032+0000", 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9250000000000003e-06 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059038+0000", 2024-04-24T14:35:23.121 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5390000000000004e-06 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.059043+0000", 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6670000000000004e-06 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072201+0000", 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013157624 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072290+0000", 2024-04-24T14:35:23.122 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9344000000000004e-05 2024-04-24T14:35:23.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:23.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.072449+0000", 2024-04-24T14:35:23.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015870599999999999 2024-04-24T14:35:23.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:23.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075280+0000", 2024-04-24T14:35:23.123 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0028318850000000001 2024-04-24T14:35:23.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.123 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.123 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:23.123 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075919+0000", 2024-04-24T14:35:23.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00063894699999999995 2024-04-24T14:35:23.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.075966+0000", 2024-04-24T14:35:23.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6721000000000002e-05 2024-04-24T14:35:23.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.124 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.124 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.124 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:32:23.076000+0000", 2024-04-24T14:35:23.124 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3787e-05 2024-04-24T14:35:23.124 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.124 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.124 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.125 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.125 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4667.0:2 3.2 3:774f7988:ns::rbd_directory:head [call rbd.dir_state_set in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-24T14:35:23.125 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:23.125 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 110.69527186800001, 2024-04-24T14:35:23.125 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.32023318200000001, 2024-04-24T14:35:23.125 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.125 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.125 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.125 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4667", 2024-04-24T14:35:23.125 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/169725836", 2024-04-24T14:35:23.125 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 2 2024-04-24T14:35:23.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.125 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:23.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365272+0000", 2024-04-24T14:35:23.126 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.126 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.126 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.126 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365269+0000", 2024-04-24T14:35:23.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:23.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365281+0000", 2024-04-24T14:35:23.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1950000000000001e-05 2024-04-24T14:35:23.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.127 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.127 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365306+0000", 2024-04-24T14:35:23.127 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4224999999999999e-05 2024-04-24T14:35:23.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.127 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365319+0000", 2024-04-24T14:35:23.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.314e-05 2024-04-24T14:35:23.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365479+0000", 2024-04-24T14:35:23.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000160607 2024-04-24T14:35:23.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.128 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.128 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:35:23.128 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.365510+0000", 2024-04-24T14:35:23.128 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0196e-05 2024-04-24T14:35:23.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.682181+0000", 2024-04-24T14:35:23.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.31667134299999999 2024-04-24T14:35:23.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.129 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.129 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685502+0000", 2024-04-24T14:35:23.129 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.003320623 2024-04-24T14:35:23.129 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.129 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.130 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.130 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:32.685505+0000", 2024-04-24T14:35:23.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7179999999999998e-06 2024-04-24T14:35:23.130 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.130 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.130 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.130 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.130 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.4772.0:40 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 4186112~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:23.130 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:23.130 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 103.00114786899999, 2024-04-24T14:35:23.130 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.033636012999999999, 2024-04-24T14:35:23.130 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.131 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.131 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.131 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.4772", 2024-04-24T14:35:23.131 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2952707455", 2024-04-24T14:35:23.131 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 40 2024-04-24T14:35:23.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.131 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.131 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.131 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:23.131 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.131 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.131 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059396+0000", 2024-04-24T14:35:23.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059394+0000", 2024-04-24T14:35:23.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:23.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.132 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.132 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.132 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059406+0000", 2024-04-24T14:35:23.132 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2323e-05 2024-04-24T14:35:23.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059438+0000", 2024-04-24T14:35:23.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.1848999999999999e-05 2024-04-24T14:35:23.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.133 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.133 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059451+0000", 2024-04-24T14:35:23.133 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2346e-05 2024-04-24T14:35:23.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059587+0000", 2024-04-24T14:35:23.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000136618 2024-04-24T14:35:23.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:23.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.059755+0000", 2024-04-24T14:35:23.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00016803900000000001 2024-04-24T14:35:23.134 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.134 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.134 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.134 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084465+0000", 2024-04-24T14:35:23.134 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.024709500999999998 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.084581+0000", 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011660800000000001 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092205+0000", 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076240429999999996 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.092505+0000", 2024-04-24T14:35:23.135 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000300067 2024-04-24T14:35:23.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.136 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.136 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.093032+0000", 2024-04-24T14:35:23.136 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00052667899999999995 2024-04-24T14:35:23.136 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.136 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.136 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.136 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.136 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 512~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:23.136 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:23.137 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 84.041787902999999, 2024-04-24T14:35:23.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045579671000000002, 2024-04-24T14:35:23.137 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.137 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.137 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.137 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:23.137 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:23.137 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 18 2024-04-24T14:35:23.137 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.137 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.137 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.137 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:23.137 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018756+0000", 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018755+0000", 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018760+0000", 2024-04-24T14:35:23.138 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7899999999999999e-06 2024-04-24T14:35:23.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018773+0000", 2024-04-24T14:35:23.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3054000000000001e-05 2024-04-24T14:35:23.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.139 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018779+0000", 2024-04-24T14:35:23.139 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9359999999999996e-06 2024-04-24T14:35:23.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.139 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.139 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019359+0000", 2024-04-24T14:35:23.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00058030500000000004 2024-04-24T14:35:23.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:23.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019435+0000", 2024-04-24T14:35:23.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5628000000000002e-05 2024-04-24T14:35:23.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.140 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.140 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.140 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045075+0000", 2024-04-24T14:35:23.140 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025639823999999999 2024-04-24T14:35:23.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:23.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045179+0000", 2024-04-24T14:35:23.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010414 2024-04-24T14:35:23.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.141 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063760+0000", 2024-04-24T14:35:23.141 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.018581569999999999 2024-04-24T14:35:23.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.141 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.141 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.063894+0000", 2024-04-24T14:35:23.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013394399999999999 2024-04-24T14:35:23.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.142 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.142 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064336+0000", 2024-04-24T14:35:23.142 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000441514 2024-04-24T14:35:23.142 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.142 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.142 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.142 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1024~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 84.041707348000003, 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.045937462999999998, 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 19 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.143 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:23.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018837+0000", 2024-04-24T14:35:23.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.144 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.144 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.144 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018836+0000", 2024-04-24T14:35:23.144 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:23.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018840+0000", 2024-04-24T14:35:23.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6269999999999999e-06 2024-04-24T14:35:23.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.145 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018850+0000", 2024-04-24T14:35:23.145 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6760000000000002e-06 2024-04-24T14:35:23.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.145 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.145 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018856+0000", 2024-04-24T14:35:23.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.9869999999999996e-06 2024-04-24T14:35:23.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.146 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019521+0000", 2024-04-24T14:35:23.146 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066467000000000002 2024-04-24T14:35:23.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.146 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.146 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:23.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019610+0000", 2024-04-24T14:35:23.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.9320999999999994e-05 2024-04-24T14:35:23.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.147 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.147 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.147 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045235+0000", 2024-04-24T14:35:23.147 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025624940999999998 2024-04-24T14:35:23.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:23.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045316+0000", 2024-04-24T14:35:23.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0610000000000002e-05 2024-04-24T14:35:23.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064416+0000", 2024-04-24T14:35:23.148 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019100076000000001 2024-04-24T14:35:23.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.148 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.148 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.148 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064567+0000", 2024-04-24T14:35:23.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000151781 2024-04-24T14:35:23.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.149 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.149 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064774+0000", 2024-04-24T14:35:23.149 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00020668100000000001 2024-04-24T14:35:23.149 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.149 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.149 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.149 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.150 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 1536~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:23.150 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:23.150 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 84.041659619000001, 2024-04-24T14:35:23.150 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046223029999999998, 2024-04-24T14:35:23.150 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.150 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.150 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.150 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:23.150 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:23.150 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 20 2024-04-24T14:35:23.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.150 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.150 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:23.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018884+0000", 2024-04-24T14:35:23.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.151 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.151 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.151 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018883+0000", 2024-04-24T14:35:23.151 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:23.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018888+0000", 2024-04-24T14:35:23.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5199999999999999e-06 2024-04-24T14:35:23.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018897+0000", 2024-04-24T14:35:23.152 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7329999999999992e-06 2024-04-24T14:35:23.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.152 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.152 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.152 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018902+0000", 2024-04-24T14:35:23.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3260000000000003e-06 2024-04-24T14:35:23.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019679+0000", 2024-04-24T14:35:23.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00077743699999999996 2024-04-24T14:35:23.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:23.153 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019747+0000", 2024-04-24T14:35:23.153 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7942999999999996e-05 2024-04-24T14:35:23.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.153 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.153 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045360+0000", 2024-04-24T14:35:23.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025612499 2024-04-24T14:35:23.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:23.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045432+0000", 2024-04-24T14:35:23.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2039999999999995e-05 2024-04-24T14:35:23.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.154 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.154 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.154 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064839+0000", 2024-04-24T14:35:23.154 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019407578000000002 2024-04-24T14:35:23.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.064954+0000", 2024-04-24T14:35:23.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000114984 2024-04-24T14:35:23.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.155 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.155 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065107+0000", 2024-04-24T14:35:23.155 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000152943 2024-04-24T14:35:23.155 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.155 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.155 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.155 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.156 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2048~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:23.156 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:23.156 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 84.041631855999995, 2024-04-24T14:35:23.156 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046523038000000003, 2024-04-24T14:35:23.156 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.156 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.156 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.156 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:23.156 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:23.156 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 21 2024-04-24T14:35:23.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.156 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.156 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:23.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018912+0000", 2024-04-24T14:35:23.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.157 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.157 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.157 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018911+0000", 2024-04-24T14:35:23.157 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:23.157 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018915+0000", 2024-04-24T14:35:23.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.58e-06 2024-04-24T14:35:23.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.158 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.158 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018934+0000", 2024-04-24T14:35:23.158 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8839999999999999e-05 2024-04-24T14:35:23.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.158 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018944+0000", 2024-04-24T14:35:23.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0485999999999999e-05 2024-04-24T14:35:23.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019806+0000", 2024-04-24T14:35:23.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00086151900000000004 2024-04-24T14:35:23.159 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.159 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.159 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:23.159 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019870+0000", 2024-04-24T14:35:23.159 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.4314999999999997e-05 2024-04-24T14:35:23.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045473+0000", 2024-04-24T14:35:23.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025602637000000001 2024-04-24T14:35:23.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.160 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:23.160 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045543+0000", 2024-04-24T14:35:23.160 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0569999999999997e-05 2024-04-24T14:35:23.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065171+0000", 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019627941999999999 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065286+0000", 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00011455699999999999 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065435+0000", 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014928900000000001 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.161 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.162 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.162 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.162 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 2560~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:23.162 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:23.162 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 84.041592421000004, 2024-04-24T14:35:23.162 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.046792357, 2024-04-24T14:35:23.162 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.162 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.162 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.162 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:23.162 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:23.163 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 22 2024-04-24T14:35:23.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.163 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:23.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.163 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.163 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.163 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018952+0000", 2024-04-24T14:35:23.163 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.163 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018951+0000", 2024-04-24T14:35:23.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:23.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018954+0000", 2024-04-24T14:35:23.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.7069999999999998e-06 2024-04-24T14:35:23.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.164 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.164 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.164 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018964+0000", 2024-04-24T14:35:23.164 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8530000000000004e-06 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018970+0000", 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2769999999999996e-06 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019928+0000", 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095859299999999999 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019996+0000", 2024-04-24T14:35:23.165 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7512e-05 2024-04-24T14:35:23.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045586+0000", 2024-04-24T14:35:23.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025590037 2024-04-24T14:35:23.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:23.166 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045652+0000", 2024-04-24T14:35:23.166 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6717999999999994e-05 2024-04-24T14:35:23.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.166 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.166 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065494+0000", 2024-04-24T14:35:23.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019841511999999999 2024-04-24T14:35:23.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065593+0000", 2024-04-24T14:35:23.167 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8759999999999996e-05 2024-04-24T14:35:23.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.167 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.167 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.167 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065744+0000", 2024-04-24T14:35:23.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015113099999999999 2024-04-24T14:35:23.168 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.168 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.168 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.168 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.168 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3072~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:23.168 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:23.168 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 84.041567436999998, 2024-04-24T14:35:23.168 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047112342000000001, 2024-04-24T14:35:23.168 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.168 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.168 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.168 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 23 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018977+0000", 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.169 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018976+0000", 2024-04-24T14:35:23.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:23.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018979+0000", 2024-04-24T14:35:23.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3629999999999998e-06 2024-04-24T14:35:23.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.170 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.170 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.170 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018990+0000", 2024-04-24T14:35:23.170 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1219999999999999e-05 2024-04-24T14:35:23.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.018996+0000", 2024-04-24T14:35:23.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.356e-06 2024-04-24T14:35:23.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.171 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.171 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020051+0000", 2024-04-24T14:35:23.171 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001055086 2024-04-24T14:35:23.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.171 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:23.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020116+0000", 2024-04-24T14:35:23.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5528e-05 2024-04-24T14:35:23.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045695+0000", 2024-04-24T14:35:23.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025578275000000001 2024-04-24T14:35:23.172 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.172 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.172 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:23.172 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045761+0000", 2024-04-24T14:35:23.172 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6509999999999998e-05 2024-04-24T14:35:23.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065802+0000", 2024-04-24T14:35:23.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020041257 2024-04-24T14:35:23.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.065912+0000", 2024-04-24T14:35:23.173 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000109833 2024-04-24T14:35:23.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.173 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.173 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.173 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066089+0000", 2024-04-24T14:35:23.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000176591 2024-04-24T14:35:23.174 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.174 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.174 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.174 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.174 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.174 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5048.0:24 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:5 [read 3584~512] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-04-24T14:35:23.174 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:23.174 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 84.041541777999996, 2024-04-24T14:35:23.174 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.047381877000000003, 2024-04-24T14:35:23.174 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.174 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.174 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.174 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5048", 2024-04-24T14:35:23.175 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1102189711", 2024-04-24T14:35:23.175 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 24 2024-04-24T14:35:23.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.175 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:23.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.175 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.175 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.175 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019002+0000", 2024-04-24T14:35:23.175 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019001+0000", 2024-04-24T14:35:23.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:23.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.176 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019006+0000", 2024-04-24T14:35:23.176 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9860000000000002e-06 2024-04-24T14:35:23.176 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.176 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.176 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019018+0000", 2024-04-24T14:35:23.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1504e-05 2024-04-24T14:35:23.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.019023+0000", 2024-04-24T14:35:23.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2399999999999998e-06 2024-04-24T14:35:23.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.177 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.177 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020154+0000", 2024-04-24T14:35:23.177 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001130523 2024-04-24T14:35:23.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.177 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for blocked object", 2024-04-24T14:35:23.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.020220+0000", 2024-04-24T14:35:23.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6068000000000005e-05 2024-04-24T14:35:23.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045789+0000", 2024-04-24T14:35:23.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.025569011999999999 2024-04-24T14:35:23.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.178 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.178 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:23.178 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.045856+0000", 2024-04-24T14:35:23.178 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6878999999999994e-05 2024-04-24T14:35:23.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066132+0000", 2024-04-24T14:35:23.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.020276272000000001 2024-04-24T14:35:23.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066238+0000", 2024-04-24T14:35:23.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105864 2024-04-24T14:35:23.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.179 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.179 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.179 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.066384+0000", 2024-04-24T14:35:23.179 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014623899999999999 2024-04-24T14:35:23.180 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.180 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.180 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.180 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.180 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.180 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:23.181 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-24T14:35:23.288 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:23.289 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:23.289 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:35:23.289 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:23.298 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-24T14:35:23.398 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:35:23.399 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:35:23.399 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:23.408 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-24T14:35:23.509 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:23.509 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:35:23.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:35:23.509 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:35:23.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.509 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:381 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1441792~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:23.509 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:23.509 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 101.84789870100001, 2024-04-24T14:35:23.509 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085854569999999995, 2024-04-24T14:35:23.509 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.510 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:23.510 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:23.510 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 381 2024-04-24T14:35:23.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.510 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.510 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.510 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:23.510 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.510 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:23.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664314+0000", 2024-04-24T14:35:23.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:23.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.511 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.511 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.511 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664317+0000", 2024-04-24T14:35:23.511 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7039999999999999e-06 2024-04-24T14:35:23.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664322+0000", 2024-04-24T14:35:23.512 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.49e-06 2024-04-24T14:35:23.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.512 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.512 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.512 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.664325+0000", 2024-04-24T14:35:23.516 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2540000000000001e-06 2024-04-24T14:35:23.516 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.516 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.516 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.671950+0000", 2024-04-24T14:35:23.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076251449999999998 2024-04-24T14:35:23.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672095+0000", 2024-04-24T14:35:23.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014422999999999999 2024-04-24T14:35:23.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.517 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.517 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.517 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.672900+0000", 2024-04-24T14:35:23.517 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00080524999999999998 2024-04-24T14:35:23.517 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.518 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.518 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.518 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.518 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:384 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1703936~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:23.518 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:23.518 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 101.84483998899999, 2024-04-24T14:35:23.518 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085404620000000004, 2024-04-24T14:35:23.518 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.518 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:23.518 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:23.519 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 384 2024-04-24T14:35:23.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.519 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:23.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.519 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.519 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:23.519 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667373+0000", 2024-04-24T14:35:23.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:23.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667375+0000", 2024-04-24T14:35:23.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4339999999999999e-06 2024-04-24T14:35:23.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.520 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.520 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.520 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667382+0000", 2024-04-24T14:35:23.520 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.6800000000000004e-06 2024-04-24T14:35:23.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667387+0000", 2024-04-24T14:35:23.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6959999999999996e-06 2024-04-24T14:35:23.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675055+0000", 2024-04-24T14:35:23.521 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076689560000000002 2024-04-24T14:35:23.521 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.521 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.521 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.521 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675201+0000", 2024-04-24T14:35:23.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000145783 2024-04-24T14:35:23.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.522 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.522 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.675914+0000", 2024-04-24T14:35:23.522 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00071240699999999997 2024-04-24T14:35:23.522 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.522 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.522 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.522 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.522 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:385 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:23.523 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:35:23.523 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 101.844822605, 2024-04-24T14:35:23.523 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0096220750000000008, 2024-04-24T14:35:23.523 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.523 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.523 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.523 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:23.523 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:23.523 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 385 2024-04-24T14:35:23.523 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.523 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.523 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.523 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:35:23.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667391+0000", 2024-04-24T14:35:23.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.524 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.524 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.524 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667390+0000", 2024-04-24T14:35:23.524 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:23.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667392+0000", 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2979999999999999e-06 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667398+0000", 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1510000000000003e-06 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.667401+0000", 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2679999999999999e-06 2024-04-24T14:35:23.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676000+0000", 2024-04-24T14:35:23.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0085992599999999992 2024-04-24T14:35:23.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.676187+0000", 2024-04-24T14:35:23.526 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018662600000000001 2024-04-24T14:35:23.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.526 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.526 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.677013+0000", 2024-04-24T14:35:23.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00082587999999999997 2024-04-24T14:35:23.527 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.527 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.527 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.527 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.527 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:400 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 3932160~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:23.527 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:35:23.527 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 101.83038651299999, 2024-04-24T14:35:23.527 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0079798239999999999, 2024-04-24T14:35:23.527 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.528 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.528 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.528 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:23.528 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:23.528 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 400 2024-04-24T14:35:23.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.528 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.528 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:35:23.528 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.528 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.528 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.528 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681827+0000", 2024-04-24T14:35:23.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681826+0000", 2024-04-24T14:35:23.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:23.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.529 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.529 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681837+0000", 2024-04-24T14:35:23.529 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1026000000000001e-05 2024-04-24T14:35:23.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.529 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681848+0000", 2024-04-24T14:35:23.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1362000000000001e-05 2024-04-24T14:35:23.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681853+0000", 2024-04-24T14:35:23.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.454e-06 2024-04-24T14:35:23.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.530 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.530 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.530 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.688973+0000", 2024-04-24T14:35:23.530 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0071199449999999999 2024-04-24T14:35:23.530 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689098+0000", 2024-04-24T14:35:23.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012527299999999999 2024-04-24T14:35:23.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.531 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.531 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689807+0000", 2024-04-24T14:35:23.531 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00070841299999999997 2024-04-24T14:35:23.531 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.531 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.531 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.532 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:401 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [sparse-read 4063232~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:23.532 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:35:23.532 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 101.830354214, 2024-04-24T14:35:23.532 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0089057549999999996, 2024-04-24T14:35:23.532 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.532 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.532 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.532 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:23.532 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:23.532 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 401 2024-04-24T14:35:23.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.532 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.532 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:35:23.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681859+0000", 2024-04-24T14:35:23.533 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.533 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.533 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.533 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.533 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681858+0000", 2024-04-24T14:35:23.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:23.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681862+0000", 2024-04-24T14:35:23.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.32e-06 2024-04-24T14:35:23.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.534 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.534 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681872+0000", 2024-04-24T14:35:23.534 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0757000000000001e-05 2024-04-24T14:35:23.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.534 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.681877+0000", 2024-04-24T14:35:23.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5310000000000003e-06 2024-04-24T14:35:23.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.689900+0000", 2024-04-24T14:35:23.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0080233509999999997 2024-04-24T14:35:23.535 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.535 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.535 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.535 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690093+0000", 2024-04-24T14:35:23.535 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019291100000000001 2024-04-24T14:35:23.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.536 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.536 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.690765+0000", 2024-04-24T14:35:23.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000671548 2024-04-24T14:35:23.536 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.536 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.536 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.536 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:416 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1835008~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:23.536 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:23.536 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 101.81607283699999, 2024-04-24T14:35:23.536 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083054860000000008, 2024-04-24T14:35:23.537 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.537 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.537 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.537 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:23.537 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:23.537 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 416 2024-04-24T14:35:23.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.537 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.537 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.537 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.537 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:23.537 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:23.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696140+0000", 2024-04-24T14:35:23.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:23.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.538 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.538 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.538 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696143+0000", 2024-04-24T14:35:23.538 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4029999999999999e-06 2024-04-24T14:35:23.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.539 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.539 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696155+0000", 2024-04-24T14:35:23.539 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1620000000000001e-05 2024-04-24T14:35:23.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.539 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.548 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.548 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696160+0000", 2024-04-24T14:35:23.548 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.9400000000000001e-06 2024-04-24T14:35:23.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703781+0000", 2024-04-24T14:35:23.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0076209670000000002 2024-04-24T14:35:23.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.703877+0000", 2024-04-24T14:35:23.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6178e-05 2024-04-24T14:35:23.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.549 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.549 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.549 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704446+0000", 2024-04-24T14:35:23.549 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00056913600000000003 2024-04-24T14:35:23.550 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.550 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.550 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.550 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:417 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [sparse-read 1966080~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:23.550 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:23.550 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 101.816046907, 2024-04-24T14:35:23.550 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093341840000000006, 2024-04-24T14:35:23.550 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.550 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.550 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.550 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:23.550 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:23.551 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 417 2024-04-24T14:35:23.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.551 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:23.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.551 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.551 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.551 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:23.551 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696166+0000", 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696169+0000", 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.191e-06 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696180+0000", 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1391e-05 2024-04-24T14:35:23.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.696185+0000", 2024-04-24T14:35:23.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8319999999999996e-06 2024-04-24T14:35:23.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704534+0000", 2024-04-24T14:35:23.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0083489210000000005 2024-04-24T14:35:23.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.553 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.553 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.553 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.704715+0000", 2024-04-24T14:35:23.553 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00018077100000000001 2024-04-24T14:35:23.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.554 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.554 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.705500+0000", 2024-04-24T14:35:23.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00078576000000000004 2024-04-24T14:35:23.554 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.554 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.554 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.554 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.554 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:432 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:23.554 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:35:23.554 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 101.800294511, 2024-04-24T14:35:23.554 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.007852777, 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 432 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.555 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711919+0000", 2024-04-24T14:35:23.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711918+0000", 2024-04-24T14:35:23.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:23.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.556 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.556 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711922+0000", 2024-04-24T14:35:23.556 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9700000000000001e-06 2024-04-24T14:35:23.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.556 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711935+0000", 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.275e-05 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711967+0000", 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2326000000000002e-05 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719755+0000", 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077882189999999999 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.557 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.558 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.558 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719771+0000", 2024-04-24T14:35:23.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6328999999999999e-05 2024-04-24T14:35:23.558 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.558 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.558 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.558 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.558 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5007.0:433 3.3 3.70b17933 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-04-24T14:35:23.558 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:35:23.558 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 101.800226995, 2024-04-24T14:35:23.558 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0078409480000000004, 2024-04-24T14:35:23.558 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.558 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5007", 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/672940643", 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 433 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711986+0000", 2024-04-24T14:35:23.559 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711985+0000", 2024-04-24T14:35:23.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:23.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.711991+0000", 2024-04-24T14:35:23.560 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.3929999999999996e-06 2024-04-24T14:35:23.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.560 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.560 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.560 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712011+0000", 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9128999999999999e-05 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.712016+0000", 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.8499999999999999e-06 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719809+0000", 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0077927430000000004 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.561 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.719827+0000", 2024-04-24T14:35:23.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8029999999999998e-05 2024-04-24T14:35:23.562 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.562 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.562 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.562 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.0.0:40 3.1 e54/18 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 54'24, mlcod=53'22)", 2024-04-24T14:35:23.562 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:23.562 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 101.742814112, 2024-04-24T14:35:23.562 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019517436999999999, 2024-04-24T14:35:23.562 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.562 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.562 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.562 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.769399+0000", 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.563 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777892+0000", 2024-04-24T14:35:23.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0084933579999999995 2024-04-24T14:35:23.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777918+0000", 2024-04-24T14:35:23.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5483999999999999e-05 2024-04-24T14:35:23.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.564 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.564 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.564 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.777940+0000", 2024-04-24T14:35:23.564 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2850999999999999e-05 2024-04-24T14:35:23.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778046+0000", 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105788 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.778080+0000", 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3521e-05 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788816+0000", 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010736173 2024-04-24T14:35:23.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.566 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.566 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:41.788917+0000", 2024-04-24T14:35:23.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100685 2024-04-24T14:35:23.566 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.566 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.566 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.566 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.566 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:103 3.1 e87/74 3:a7f4e17d:::rbd_data.12ff64619e2d.0000000000000004:head v 87'261, mlcod=87'258)", 2024-04-24T14:35:23.566 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:35:23.566 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 56.551532907999999, 2024-04-24T14:35:23.566 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019745728000000001, 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960680+0000", 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.567 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960679+0000", 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960692+0000", 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3709999999999999e-05 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960722+0000", 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9791999999999999e-05 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.568 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960733+0000", 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.128e-05 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960857+0000", 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000123981 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.960926+0000", 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8881999999999995e-05 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.569 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980353+0000", 2024-04-24T14:35:23.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019426853000000001 2024-04-24T14:35:23.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.570 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.570 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980426+0000", 2024-04-24T14:35:23.570 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.2803000000000001e-05 2024-04-24T14:35:23.570 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.570 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.570 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.570 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:105 3.1 e87/74 3:bf9ee08e:::rbd_data.12ff64619e2d.0000000000000005:head v 87'263, mlcod=87'258)", 2024-04-24T14:35:23.570 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 56.542406374000002, 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.014864743999999999, 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969807+0000", 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969805+0000", 2024-04-24T14:35:23.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:23.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969820+0000", 2024-04-24T14:35:23.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5116999999999999e-05 2024-04-24T14:35:23.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.572 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.572 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.572 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969852+0000", 2024-04-24T14:35:23.572 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2135999999999999e-05 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.969864+0000", 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.189e-05 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972122+0000", 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022580389999999999 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.972165+0000", 2024-04-24T14:35:23.573 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2429000000000003e-05 2024-04-24T14:35:23.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984619+0000", 2024-04-24T14:35:23.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012454735 2024-04-24T14:35:23.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.574 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.574 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.574 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984672+0000", 2024-04-24T14:35:23.574 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2077999999999998e-05 2024-04-24T14:35:23.574 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.574 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.574 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:112 3.1 e87/74 3:a983798d:::rbd_data.12ff64619e2d.0000000000000006:head v 87'265, mlcod=87'258)", 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 56.532067538, 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012821978, 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.575 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980146+0000", 2024-04-24T14:35:23.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980144+0000", 2024-04-24T14:35:23.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:23.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.576 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.576 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.576 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980159+0000", 2024-04-24T14:35:23.576 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4769999999999999e-05 2024-04-24T14:35:23.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980189+0000", 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.0419000000000001e-05 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980213+0000", 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4036000000000002e-05 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980452+0000", 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00023921500000000001 2024-04-24T14:35:23.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.980468+0000", 2024-04-24T14:35:23.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5843999999999999e-05 2024-04-24T14:35:23.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992932+0000", 2024-04-24T14:35:23.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.012463314 2024-04-24T14:35:23.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.578 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.578 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.578 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992968+0000", 2024-04-24T14:35:23.578 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6075999999999998e-05 2024-04-24T14:35:23.579 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.579 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.579 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.579 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:113 3.1 e87/74 3:a462e6bd:::rbd_data.12ff64619e2d.0000000000000007:head v 87'267, mlcod=87'258)", 2024-04-24T14:35:23.579 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:35:23.579 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 56.528532814000002, 2024-04-24T14:35:23.579 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010561022999999999, 2024-04-24T14:35:23.579 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.579 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.579 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983680+0000", 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983679+0000", 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983699+0000", 2024-04-24T14:35:23.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0392000000000001e-05 2024-04-24T14:35:23.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983737+0000", 2024-04-24T14:35:23.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8340000000000002e-05 2024-04-24T14:35:23.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.581 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.581 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.581 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.983756+0000", 2024-04-24T14:35:23.581 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8402000000000001e-05 2024-04-24T14:35:23.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984706+0000", 2024-04-24T14:35:23.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00095024400000000005 2024-04-24T14:35:23.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984741+0000", 2024-04-24T14:35:23.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4721000000000002e-05 2024-04-24T14:35:23.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.582 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.582 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.582 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994211+0000", 2024-04-24T14:35:23.582 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0094708559999999997 2024-04-24T14:35:23.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.583 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.583 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994241+0000", 2024-04-24T14:35:23.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9941e-05 2024-04-24T14:35:23.583 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.583 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.583 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.583 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.583 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:115 3.3 3:cefcbaf8:::rbd_data.12ff64619e2d.0000000000000009:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:23.583 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:35:23.583 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 56.528078776999998, 2024-04-24T14:35:23.583 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010347242, 2024-04-24T14:35:23.584 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.584 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:35:23.584 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:23.584 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 115 2024-04-24T14:35:23.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.584 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.584 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.584 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:35:23.584 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.584 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984134+0000", 2024-04-24T14:35:23.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984132+0000", 2024-04-24T14:35:23.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:23.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.585 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.585 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.585 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984142+0000", 2024-04-24T14:35:23.585 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.7370000000000001e-06 2024-04-24T14:35:23.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984154+0000", 2024-04-24T14:35:23.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2288e-05 2024-04-24T14:35:23.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984163+0000", 2024-04-24T14:35:23.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.3749999999999992e-06 2024-04-24T14:35:23.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.586 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.586 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.586 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984246+0000", 2024-04-24T14:35:23.586 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2962999999999998e-05 2024-04-24T14:35:23.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984473+0000", 2024-04-24T14:35:23.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00022696399999999999 2024-04-24T14:35:23.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:23.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.984771+0000", 2024-04-24T14:35:23.587 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00029739499999999999 2024-04-24T14:35:23.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.587 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.587 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:23.587 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.992998+0000", 2024-04-24T14:35:23.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0082275750000000009 2024-04-24T14:35:23.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:23.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994299+0000", 2024-04-24T14:35:23.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013005289999999999 2024-04-24T14:35:23.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.588 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994403+0000", 2024-04-24T14:35:23.588 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00010376499999999999 2024-04-24T14:35:23.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.588 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.588 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.589 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:26.994482+0000", 2024-04-24T14:35:23.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9013999999999999e-05 2024-04-24T14:35:23.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.589 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.589 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5324.0:128 3.3 3:cc9e8d0e:::rbd_data.12ff64619e2d.000000000000000a:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:23.589 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:35:23.589 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 56.506887849000002, 2024-04-24T14:35:23.589 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011784144, 2024-04-24T14:35:23.589 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.589 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.589 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.590 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5324", 2024-04-24T14:35:23.590 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:23.590 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 128 2024-04-24T14:35:23.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.590 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:35:23.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.590 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.590 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005325+0000", 2024-04-24T14:35:23.590 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005324+0000", 2024-04-24T14:35:23.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:23.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005332+0000", 2024-04-24T14:35:23.591 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.2989999999999997e-06 2024-04-24T14:35:23.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.591 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.591 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.591 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005354+0000", 2024-04-24T14:35:23.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1705999999999999e-05 2024-04-24T14:35:23.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005362+0000", 2024-04-24T14:35:23.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.5599999999999996e-06 2024-04-24T14:35:23.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.592 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.592 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005418+0000", 2024-04-24T14:35:23.592 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6685999999999999e-05 2024-04-24T14:35:23.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.592 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005535+0000", 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000117008 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005732+0000", 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00019718700000000001 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016440+0000", 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010707787999999999 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.593 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:23.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.016913+0000", 2024-04-24T14:35:23.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00047270400000000001 2024-04-24T14:35:23.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017047+0000", 2024-04-24T14:35:23.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00013434699999999999 2024-04-24T14:35:23.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.594 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.594 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.594 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.017109+0000", 2024-04-24T14:35:23.594 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.2192000000000004e-05 2024-04-24T14:35:23.595 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.597 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.597 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.597 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.597 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.5324.0:130 3.1 e87/74 3:b0d79d63:::rbd_data.12ff64619e2d.000000000000000c:head v 87'269, mlcod=87'265)", 2024-04-24T14:35:23.598 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:35:23.598 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 56.506435089, 2024-04-24T14:35:23.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013945631999999999, 2024-04-24T14:35:23.598 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.598 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.598 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:35:23.598 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.598 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.598 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.598 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005778+0000", 2024-04-24T14:35:23.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005777+0000", 2024-04-24T14:35:23.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:23.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.599 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005791+0000", 2024-04-24T14:35:23.599 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3743999999999999e-05 2024-04-24T14:35:23.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.599 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005803+0000", 2024-04-24T14:35:23.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2330999999999999e-05 2024-04-24T14:35:23.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005810+0000", 2024-04-24T14:35:23.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8990000000000004e-06 2024-04-24T14:35:23.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.600 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.600 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005888+0000", 2024-04-24T14:35:23.600 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7872000000000003e-05 2024-04-24T14:35:23.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.600 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.005911+0000", 2024-04-24T14:35:23.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3156e-05 2024-04-24T14:35:23.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019642+0000", 2024-04-24T14:35:23.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013730456 2024-04-24T14:35:23.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.601 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.601 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.601 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:27.019724+0000", 2024-04-24T14:35:23.601 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.1981e-05 2024-04-24T14:35:23.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.602 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.602 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.602 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.2.0:211 3.0 e116/99 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:7 v 116'270, mlcod=115'266)", 2024-04-24T14:35:23.602 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:17.397582+0000", 2024-04-24T14:35:23.602 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.1146315930000004, 2024-04-24T14:35:23.602 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011810169000000001, 2024-04-24T14:35:23.602 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.602 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.602 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.602 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.602 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.602 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397582+0000", 2024-04-24T14:35:23.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397582+0000", 2024-04-24T14:35:23.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.603 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397581+0000", 2024-04-24T14:35:23.603 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:23.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.603 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.603 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397596+0000", 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.485e-05 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397625+0000", 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9892000000000001e-05 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397647+0000", 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1733e-05 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.604 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397773+0000", 2024-04-24T14:35:23.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000126277 2024-04-24T14:35:23.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.397850+0000", 2024-04-24T14:35:23.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6339000000000002e-05 2024-04-24T14:35:23.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.605 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.605 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.605 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.409320+0000", 2024-04-24T14:35:23.605 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011470556 2024-04-24T14:35:23.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.606 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.606 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.409392+0000", 2024-04-24T14:35:23.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.1471999999999996e-05 2024-04-24T14:35:23.606 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.606 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.606 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.606 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.606 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5795.0:56 3.1 3:990e5750:::rbd_header.12ff64619e2d:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e116)", 2024-04-24T14:35:23.606 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:17.399479+0000", 2024-04-24T14:35:23.606 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.1127343659999998, 2024-04-24T14:35:23.606 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.013769491, 2024-04-24T14:35:23.607 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.607 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:23.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5795", 2024-04-24T14:35:23.607 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1567306440", 2024-04-24T14:35:23.607 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 56 2024-04-24T14:35:23.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.607 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.607 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.607 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399479+0000", 2024-04-24T14:35:23.607 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.607 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399479+0000", 2024-04-24T14:35:23.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399476+0000", 2024-04-24T14:35:23.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:35:23.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.608 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.608 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.608 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399491+0000", 2024-04-24T14:35:23.608 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5688999999999998e-05 2024-04-24T14:35:23.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399527+0000", 2024-04-24T14:35:23.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6192999999999997e-05 2024-04-24T14:35:23.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399547+0000", 2024-04-24T14:35:23.609 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9156000000000001e-05 2024-04-24T14:35:23.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.609 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.609 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.609 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399650+0000", 2024-04-24T14:35:23.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000103346 2024-04-24T14:35:23.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op must wait for map", 2024-04-24T14:35:23.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.399663+0000", 2024-04-24T14:35:23.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.345e-05 2024-04-24T14:35:23.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.610 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.402177+0000", 2024-04-24T14:35:23.610 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025136659999999999 2024-04-24T14:35:23.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.610 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.610 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.402336+0000", 2024-04-24T14:35:23.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00015912700000000001 2024-04-24T14:35:23.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 0", 2024-04-24T14:35:23.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.402739+0000", 2024-04-24T14:35:23.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000402736 2024-04-24T14:35:23.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.611 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:23.611 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.413101+0000", 2024-04-24T14:35:23.611 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.010361746 2024-04-24T14:35:23.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.611 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.413152+0000", 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1515999999999999e-05 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.413206+0000", 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.4316e-05 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.413248+0000", 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1820000000000003e-05 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.612 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.613 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.2.0:212 3.0 e116/99 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:7 v 116'272, mlcod=115'266)", 2024-04-24T14:35:23.613 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:17.401237+0000", 2024-04-24T14:35:23.613 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 6.1109760729999998, 2024-04-24T14:35:23.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011174969999999999, 2024-04-24T14:35:23.613 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.613 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.613 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.613 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.613 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.401237+0000", 2024-04-24T14:35:23.613 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.401237+0000", 2024-04-24T14:35:23.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.401235+0000", 2024-04-24T14:35:23.614 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:23.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.614 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.614 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.401276+0000", 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.1009e-05 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.401321+0000", 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4778999999999999e-05 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.401337+0000", 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5807e-05 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.615 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.403013+0000", 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016755909999999999 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.403037+0000", 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4485e-05 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.412346+0000", 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0093090180000000005 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.616 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:17.412412+0000", 2024-04-24T14:35:23.617 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.5983999999999997e-05 2024-04-24T14:35:23.617 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.617 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.617 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.617 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.617 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.617 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:23.618 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-24T14:35:23.725 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:23.725 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:23.725 INFO:teuthology.orchestra.run.smithi103.stdout: "num_ops": 0 2024-04-24T14:35:23.725 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:23.735 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-24T14:35:23.836 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:23.836 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [], 2024-04-24T14:35:23.836 INFO:teuthology.orchestra.run.smithi103.stdout: "complaint_time": 30, 2024-04-24T14:35:23.836 INFO:teuthology.orchestra.run.smithi103.stdout: "num_blocked_ops": 0 2024-04-24T14:35:23.836 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:23.846 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-24T14:35:23.946 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-04-24T14:35:23.946 INFO:teuthology.orchestra.run.smithi103.stdout: "size": 20, 2024-04-24T14:35:23.946 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 600, 2024-04-24T14:35:23.946 INFO:teuthology.orchestra.run.smithi103.stdout: "ops": [ 2024-04-24T14:35:23.946 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.946 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:144 3.2 e53/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 53'13, mlcod=49'11)", 2024-04-24T14:35:23.947 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:23.947 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 103.877678438, 2024-04-24T14:35:23.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.019782988000000001, 2024-04-24T14:35:23.947 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.947 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.947 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.947 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:23.947 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.947 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.947 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071932+0000", 2024-04-24T14:35:23.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.071929+0000", 2024-04-24T14:35:23.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999976 2024-04-24T14:35:23.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.948 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.948 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076271+0000", 2024-04-24T14:35:23.948 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043416050000000001 2024-04-24T14:35:23.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.948 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076316+0000", 2024-04-24T14:35:23.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.4802000000000002e-05 2024-04-24T14:35:23.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076336+0000", 2024-04-24T14:35:23.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9754000000000001e-05 2024-04-24T14:35:23.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.949 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.949 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.949 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076422+0000", 2024-04-24T14:35:23.949 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6553999999999998e-05 2024-04-24T14:35:23.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.076475+0000", 2024-04-24T14:35:23.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.3115999999999998e-05 2024-04-24T14:35:23.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091638+0000", 2024-04-24T14:35:23.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015163183 2024-04-24T14:35:23.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.950 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.950 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:40.091715+0000", 2024-04-24T14:35:23.950 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.6290000000000003e-05 2024-04-24T14:35:23.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.951 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.951 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:53 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'52, mlcod=57'50)", 2024-04-24T14:35:23.951 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:23.951 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 89.882863858999997, 2024-04-24T14:35:23.951 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043819260000000004, 2024-04-24T14:35:23.951 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.951 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.951 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.951 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:23.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066746+0000", 2024-04-24T14:35:23.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.952 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.952 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.066744+0000", 2024-04-24T14:35:23.952 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068071+0000", 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0013269670000000001 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068119+0000", 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7922e-05 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068136+0000", 2024-04-24T14:35:23.953 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.7263000000000001e-05 2024-04-24T14:35:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068257+0000", 2024-04-24T14:35:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012023099999999999 2024-04-24T14:35:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.068314+0000", 2024-04-24T14:35:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.7732000000000002e-05 2024-04-24T14:35:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.954 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071090+0000", 2024-04-24T14:35:23.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027752499999999999 2024-04-24T14:35:23.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.955 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.955 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.071128+0000", 2024-04-24T14:35:23.955 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8587999999999997e-05 2024-04-24T14:35:23.955 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.955 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.955 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.955 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.955 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:55 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'54, mlcod=62'52)", 2024-04-24T14:35:23.955 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:23.955 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 89.871934601000007, 2024-04-24T14:35:23.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049146830000000004, 2024-04-24T14:35:23.956 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.956 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.956 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:23.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.956 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.956 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.956 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077676+0000", 2024-04-24T14:35:23.956 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.077674+0000", 2024-04-24T14:35:23.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:23.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078083+0000", 2024-04-24T14:35:23.957 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040933300000000001 2024-04-24T14:35:23.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.957 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.957 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.957 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078105+0000", 2024-04-24T14:35:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2065999999999998e-05 2024-04-24T14:35:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078115+0000", 2024-04-24T14:35:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.2769999999999994e-06 2024-04-24T14:35:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078195+0000", 2024-04-24T14:35:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.0012999999999994e-05 2024-04-24T14:35:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.958 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.078231+0000", 2024-04-24T14:35:23.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.6698e-05 2024-04-24T14:35:23.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082507+0000", 2024-04-24T14:35:23.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0042756160000000003 2024-04-24T14:35:23.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.959 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.959 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.959 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.082590+0000", 2024-04-24T14:35:23.959 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.3251999999999995e-05 2024-04-24T14:35:23.959 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.959 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.960 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.960 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:57 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'56, mlcod=62'54)", 2024-04-24T14:35:23.960 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:23.960 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 89.860978520000003, 2024-04-24T14:35:23.960 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0049506919999999996, 2024-04-24T14:35:23.960 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.960 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.960 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.960 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.960 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.960 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:23.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088632+0000", 2024-04-24T14:35:23.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.961 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.088630+0000", 2024-04-24T14:35:23.961 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:23.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.961 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.961 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089035+0000", 2024-04-24T14:35:23.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00040466599999999998 2024-04-24T14:35:23.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089056+0000", 2024-04-24T14:35:23.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.1001000000000001e-05 2024-04-24T14:35:23.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.962 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.962 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089076+0000", 2024-04-24T14:35:23.962 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.9885999999999999e-05 2024-04-24T14:35:23.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.962 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089156+0000", 2024-04-24T14:35:23.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9944999999999994e-05 2024-04-24T14:35:23.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.089202+0000", 2024-04-24T14:35:23.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6356999999999997e-05 2024-04-24T14:35:23.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.963 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.963 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093515+0000", 2024-04-24T14:35:23.963 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0043128039999999999 2024-04-24T14:35:23.963 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.964 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.964 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.093582+0000", 2024-04-24T14:35:23.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7193000000000005e-05 2024-04-24T14:35:23.964 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.964 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.964 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.964 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.964 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:59 3.2 e62/46 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head v 62'58, mlcod=62'56)", 2024-04-24T14:35:23.964 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:23.964 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 89.849409155000004, 2024-04-24T14:35:23.964 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0060813769999999998, 2024-04-24T14:35:23.964 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.965 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.965 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:23.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.965 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.965 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100201+0000", 2024-04-24T14:35:23.965 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.965 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.965 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100199+0000", 2024-04-24T14:35:23.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:23.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100792+0000", 2024-04-24T14:35:23.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00059307100000000003 2024-04-24T14:35:23.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.966 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.966 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.966 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100831+0000", 2024-04-24T14:35:23.966 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.8362000000000003e-05 2024-04-24T14:35:23.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100847+0000", 2024-04-24T14:35:23.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6147999999999999e-05 2024-04-24T14:35:23.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.100949+0000", 2024-04-24T14:35:23.967 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000101897 2024-04-24T14:35:23.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.967 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.967 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.967 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.101009+0000", 2024-04-24T14:35:23.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.0470000000000002e-05 2024-04-24T14:35:23.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106184+0000", 2024-04-24T14:35:23.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0051743809999999996 2024-04-24T14:35:23.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.968 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.968 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.968 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.106282+0000", 2024-04-24T14:35:23.968 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8820000000000006e-05 2024-04-24T14:35:23.968 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.968 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.968 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:111 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'62, mlcod=62'60)", 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 89.596289495999997, 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004526937, 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.969 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353321+0000", 2024-04-24T14:35:23.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353319+0000", 2024-04-24T14:35:23.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:23.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.970 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.970 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.970 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353824+0000", 2024-04-24T14:35:23.970 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00050414999999999998 2024-04-24T14:35:23.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353850+0000", 2024-04-24T14:35:23.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6480999999999999e-05 2024-04-24T14:35:23.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353861+0000", 2024-04-24T14:35:23.971 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0596e-05 2024-04-24T14:35:23.971 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.971 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.971 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.971 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353959+0000", 2024-04-24T14:35:23.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.8487999999999996e-05 2024-04-24T14:35:23.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.353999+0000", 2024-04-24T14:35:23.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.0099999999999999e-05 2024-04-24T14:35:23.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.972 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357796+0000", 2024-04-24T14:35:23.972 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0037964230000000002 2024-04-24T14:35:23.972 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.972 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.972 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.973 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.357848+0000", 2024-04-24T14:35:23.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.1978999999999998e-05 2024-04-24T14:35:23.973 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.973 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.973 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.973 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.973 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:113 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'64, mlcod=62'62)", 2024-04-24T14:35:23.973 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:23.973 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 89.585544327999997, 2024-04-24T14:35:23.973 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0058741460000000002, 2024-04-24T14:35:23.973 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.973 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.973 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:23.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.974 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364066+0000", 2024-04-24T14:35:23.974 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.974 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.974 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364064+0000", 2024-04-24T14:35:23.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:23.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364669+0000", 2024-04-24T14:35:23.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00060488800000000004 2024-04-24T14:35:23.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.975 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.975 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364708+0000", 2024-04-24T14:35:23.975 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.9595999999999998e-05 2024-04-24T14:35:23.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.975 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364725+0000", 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6725999999999999e-05 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364812+0000", 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.6637000000000004e-05 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.364859+0000", 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7252000000000001e-05 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.976 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369871+0000", 2024-04-24T14:35:23.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050118380000000002 2024-04-24T14:35:23.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.977 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.977 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.369940+0000", 2024-04-24T14:35:23.977 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9053000000000004e-05 2024-04-24T14:35:23.977 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.977 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.977 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.977 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.977 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.978 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:115 3.2 e62/46 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head v 62'66, mlcod=62'64)", 2024-04-24T14:35:23.978 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:23.978 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 89.573178052000003, 2024-04-24T14:35:23.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0052176540000000004, 2024-04-24T14:35:23.978 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.978 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.978 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.978 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:23.978 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.978 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.978 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376432+0000", 2024-04-24T14:35:23.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376431+0000", 2024-04-24T14:35:23.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:23.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.979 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.979 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376916+0000", 2024-04-24T14:35:23.979 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00048492000000000001 2024-04-24T14:35:23.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.979 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376949+0000", 2024-04-24T14:35:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.3691e-05 2024-04-24T14:35:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.376964+0000", 2024-04-24T14:35:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.4307999999999999e-05 2024-04-24T14:35:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377035+0000", 2024-04-24T14:35:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.0949000000000001e-05 2024-04-24T14:35:23.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.377064+0000", 2024-04-24T14:35:23.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9456999999999999e-05 2024-04-24T14:35:23.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381510+0000", 2024-04-24T14:35:23.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004446027 2024-04-24T14:35:23.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.981 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.981 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.981 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.381650+0000", 2024-04-24T14:35:23.981 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000139727 2024-04-24T14:35:23.982 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.982 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.982 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.982 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.982 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:142 3.2 e62/46 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head v 62'74, mlcod=62'72)", 2024-04-24T14:35:23.982 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:23.982 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 89.419126245000001, 2024-04-24T14:35:23.982 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047663150000000001, 2024-04-24T14:35:23.982 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.982 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.982 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.982 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:23.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530484+0000", 2024-04-24T14:35:23.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.983 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.983 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530483+0000", 2024-04-24T14:35:23.983 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:23.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.983 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530850+0000", 2024-04-24T14:35:23.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00036732699999999997 2024-04-24T14:35:23.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530874+0000", 2024-04-24T14:35:23.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.4457999999999999e-05 2024-04-24T14:35:23.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.984 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.984 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.984 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530887+0000", 2024-04-24T14:35:23.984 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2896e-05 2024-04-24T14:35:23.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530955+0000", 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.7161000000000003e-05 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.530990+0000", 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.5280000000000001e-05 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535166+0000", 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041761960000000001 2024-04-24T14:35:23.985 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.986 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.986 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.535250+0000", 2024-04-24T14:35:23.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.4271999999999995e-05 2024-04-24T14:35:23.986 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.986 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.986 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.986 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.986 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(client.4937.0:176 3.2 e62/46 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head v 62'80, mlcod=62'78)", 2024-04-24T14:35:23.986 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:23.986 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 89.249287569000003, 2024-04-24T14:35:23.986 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0047095959999999999, 2024-04-24T14:35:23.987 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.987 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.987 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:23.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.987 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.987 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700323+0000", 2024-04-24T14:35:23.987 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.987 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700321+0000", 2024-04-24T14:35:23.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:23.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700834+0000", 2024-04-24T14:35:23.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00051365999999999998 2024-04-24T14:35:23.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.988 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.988 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.988 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700857+0000", 2024-04-24T14:35:23.988 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2733000000000001e-05 2024-04-24T14:35:23.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700865+0000", 2024-04-24T14:35:23.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.7400000000000004e-06 2024-04-24T14:35:23.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.700961+0000", 2024-04-24T14:35:23.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.6045000000000006e-05 2024-04-24T14:35:23.989 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.989 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.989 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.989 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.701031+0000", 2024-04-24T14:35:23.989 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.9740999999999995e-05 2024-04-24T14:35:23.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.704976+0000", 2024-04-24T14:35:23.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0039450040000000002 2024-04-24T14:35:23.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.990 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.990 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.990 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:54.705032+0000", 2024-04-24T14:35:23.990 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.6644999999999997e-05 2024-04-24T14:35:23.990 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.990 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.990 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.991 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_repop(osd.3.0:215 3.2 e64/46 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head v 64'83, mlcod=62'81)", 2024-04-24T14:35:23.991 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:23.991 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 84.914316733999996, 2024-04-24T14:35:23.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.015919574999999998, 2024-04-24T14:35:23.991 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.991 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:23.991 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.991 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.991 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.991 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:23.991 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:23.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.035293+0000", 2024-04-24T14:35:23.992 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:23.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.992 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.992 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.992 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039394+0000", 2024-04-24T14:35:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0041011800000000003 2024-04-24T14:35:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039423+0000", 2024-04-24T14:35:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.9388999999999999e-05 2024-04-24T14:35:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039438+0000", 2024-04-24T14:35:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.464e-05 2024-04-24T14:35:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.993 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039491+0000", 2024-04-24T14:35:23.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2858000000000001e-05 2024-04-24T14:35:23.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:23.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.039518+0000", 2024-04-24T14:35:23.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7614e-05 2024-04-24T14:35:23.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.994 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.994 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:23.994 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051105+0000", 2024-04-24T14:35:23.994 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.011586953000000001 2024-04-24T14:35:23.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.995 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.995 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:33:59.051213+0000", 2024-04-24T14:35:23.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000107868 2024-04-24T14:35:23.995 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.995 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.995 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.995 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.995 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.995 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPull(3.2 e68/67 cost 1048576)", 2024-04-24T14:35:23.995 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:23.995 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 81.070165986000006, 2024-04-24T14:35:23.995 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055824159999999998, 2024-04-24T14:35:23.996 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.996 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "reached pg", 2024-04-24T14:35:23.996 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:23.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:23.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.996 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:23.996 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:23.996 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:23.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.996 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:23.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879444+0000", 2024-04-24T14:35:23.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:23.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:23.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879459+0000", 2024-04-24T14:35:23.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.5075000000000001e-05 2024-04-24T14:35:23.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.997 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.997 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:23.997 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879485+0000", 2024-04-24T14:35:23.997 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.6353e-05 2024-04-24T14:35:23.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:23.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879493+0000", 2024-04-24T14:35:23.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.9789999999999993e-06 2024-04-24T14:35:23.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:23.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.879535+0000", 2024-04-24T14:35:23.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.2585e-05 2024-04-24T14:35:23.998 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.998 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.998 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:23.998 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:02.885027+0000", 2024-04-24T14:35:23.998 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0054911099999999996 2024-04-24T14:35:23.999 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.999 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:23.999 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:23.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:23.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.999 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:35:23.999 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:23.999 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.846402986000001, 2024-04-24T14:35:23.999 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044084859999999997, 2024-04-24T14:35:23.999 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:23.999 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:23.999 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:23.999 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:23.999 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103207+0000", 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.103206+0000", 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.000 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:24.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104731+0000", 2024-04-24T14:35:24.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001524755 2024-04-24T14:35:24.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:24.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104759+0000", 2024-04-24T14:35:24.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.7973000000000001e-05 2024-04-24T14:35:24.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.001 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.001 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104807+0000", 2024-04-24T14:35:24.001 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.8112000000000003e-05 2024-04-24T14:35:24.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.001 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:24.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104907+0000", 2024-04-24T14:35:24.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000100317 2024-04-24T14:35:24.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:24.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.104918+0000", 2024-04-24T14:35:24.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.0747999999999999e-05 2024-04-24T14:35:24.002 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.002 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.002 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:24.002 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.107616+0000", 2024-04-24T14:35:24.002 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0026977559999999999 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 83/82 [PushOp(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head, version: 49'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head@49'2, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 65.823108032999997, 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0046193859999999996, 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:24.003 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:24.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:24.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126502+0000", 2024-04-24T14:35:24.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:24.004 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.126501+0000", 2024-04-24T14:35:24.004 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:24.004 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.004 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.004 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:24.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128169+0000", 2024-04-24T14:35:24.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0016686229999999999 2024-04-24T14:35:24.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:24.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128202+0000", 2024-04-24T14:35:24.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2539000000000001e-05 2024-04-24T14:35:24.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.005 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.005 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128222+0000", 2024-04-24T14:35:24.005 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0149e-05 2024-04-24T14:35:24.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.005 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:24.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128375+0000", 2024-04-24T14:35:24.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000153459 2024-04-24T14:35:24.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:24.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.128399+0000", 2024-04-24T14:35:24.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.3393000000000001e-05 2024-04-24T14:35:24.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.006 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.006 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:24.006 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:34:18.131122+0000", 2024-04-24T14:35:24.006 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0027227140000000002 2024-04-24T14:35:24.007 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.007 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:24.007 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.007 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.007 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.007 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5674.0:27 3.2 3:5592b7cf:::journal.12ff64619e2d:head [watch watch cookie 139908626846656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-04-24T14:35:24.007 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.218271+0000", 2024-04-24T14:35:24.007 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.731339439999999, 2024-04-24T14:35:24.007 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.107948372, 2024-04-24T14:35:24.007 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:24.007 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:24.007 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:24.007 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5674", 2024-04-24T14:35:24.007 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1498157813", 2024-04-24T14:35:24.008 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 27 2024-04-24T14:35:24.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.008 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:24.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:24.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218271+0000", 2024-04-24T14:35:24.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.008 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:24.008 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218271+0000", 2024-04-24T14:35:24.008 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.008 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218267+0000", 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999967 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218280+0000", 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.2469e-05 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218303+0000", 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.2952999999999999e-05 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.009 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218316+0000", 2024-04-24T14:35:24.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.313e-05 2024-04-24T14:35:24.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:24.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218446+0000", 2024-04-24T14:35:24.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000130154 2024-04-24T14:35:24.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.010 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for peered", 2024-04-24T14:35:24.010 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.218460+0000", 2024-04-24T14:35:24.010 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.3852999999999999e-05 2024-04-24T14:35:24.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:24.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.320512+0000", 2024-04-24T14:35:24.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.102051709 2024-04-24T14:35:24.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for missing object", 2024-04-24T14:35:24.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.320660+0000", 2024-04-24T14:35:24.011 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000148362 2024-04-24T14:35:24.011 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.011 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.011 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:24.011 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.322855+0000", 2024-04-24T14:35:24.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0021955249999999998 2024-04-24T14:35:24.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:24.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.322976+0000", 2024-04-24T14:35:24.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000120813 2024-04-24T14:35:24.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:24.012 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.323702+0000", 2024-04-24T14:35:24.012 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00072575199999999997 2024-04-24T14:35:24.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.012 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.012 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:24.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.323827+0000", 2024-04-24T14:35:24.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00012478200000000001 2024-04-24T14:35:24.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:24.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.324040+0000", 2024-04-24T14:35:24.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00021278700000000001 2024-04-24T14:35:24.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.013 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:24.013 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.324936+0000", 2024-04-24T14:35:24.013 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00089639600000000004 2024-04-24T14:35:24.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.013 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:24.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.326067+0000", 2024-04-24T14:35:24.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0011309110000000001 2024-04-24T14:35:24.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:24.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.326114+0000", 2024-04-24T14:35:24.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.6903999999999997e-05 2024-04-24T14:35:24.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.014 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.014 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:24.014 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.326219+0000", 2024-04-24T14:35:24.014 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000105368 2024-04-24T14:35:24.015 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.015 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:24.015 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.015 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 108/107 [PushOp(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head, version: 49'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head@49'1, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-24T14:35:24.015 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.396664+0000", 2024-04-24T14:35:24.015 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.552946521999999, 2024-04-24T14:35:24.015 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.004962241, 2024-04-24T14:35:24.015 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:24.015 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:24.015 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:24.015 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:24.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.396664+0000", 2024-04-24T14:35:24.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:24.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.396664+0000", 2024-04-24T14:35:24.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.016 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.016 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:24.016 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.396662+0000", 2024-04-24T14:35:24.016 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:24.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:24.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.398018+0000", 2024-04-24T14:35:24.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001355904 2024-04-24T14:35:24.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:24.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.398037+0000", 2024-04-24T14:35:24.017 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.8907e-05 2024-04-24T14:35:24.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.017 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.017 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.017 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.398049+0000", 2024-04-24T14:35:24.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.1661000000000001e-05 2024-04-24T14:35:24.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:24.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.400590+0000", 2024-04-24T14:35:24.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025413749999999998 2024-04-24T14:35:24.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.018 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:24.018 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.400599+0000", 2024-04-24T14:35:24.018 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 9.0420000000000008e-06 2024-04-24T14:35:24.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.018 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.019 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:24.019 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.401626+0000", 2024-04-24T14:35:24.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0010267119999999999 2024-04-24T14:35:24.019 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.019 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:24.019 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.019 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.019 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(osd.5.11:8 2.2 2:4f9137c5:::journal_data.2.12ff64619e2d.3:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e108)", 2024-04-24T14:35:24.019 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.439647+0000", 2024-04-24T14:35:24.019 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.509962882, 2024-04-24T14:35:24.019 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0050746209999999996, 2024-04-24T14:35:24.019 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:24.019 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:24.020 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:24.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:24.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439647+0000", 2024-04-24T14:35:24.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:24.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439647+0000", 2024-04-24T14:35:24.020 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.020 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:24.020 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439646+0000", 2024-04-24T14:35:24.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.999999 2024-04-24T14:35:24.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:24.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439652+0000", 2024-04-24T14:35:24.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.0170000000000004e-06 2024-04-24T14:35:24.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.021 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:24.021 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439660+0000", 2024-04-24T14:35:24.021 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 8.7730000000000005e-06 2024-04-24T14:35:24.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.021 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.439665+0000", 2024-04-24T14:35:24.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.5220000000000001e-06 2024-04-24T14:35:24.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:24.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.441882+0000", 2024-04-24T14:35:24.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0022169260000000001 2024-04-24T14:35:24.022 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.022 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.022 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for rw locks", 2024-04-24T14:35:24.022 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.442038+0000", 2024-04-24T14:35:24.022 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000156661 2024-04-24T14:35:24.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:24.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.443062+0000", 2024-04-24T14:35:24.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001023604 2024-04-24T14:35:24.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:24.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.443281+0000", 2024-04-24T14:35:24.023 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000218564 2024-04-24T14:35:24.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.023 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.023 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:24.023 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.444722+0000", 2024-04-24T14:35:24.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0014413539999999999 2024-04-24T14:35:24.024 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.024 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:24.024 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.024 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.024 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5674.0:200 3.2 3:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head [read 3895296~36864] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-04-24T14:35:24.024 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.470076+0000", 2024-04-24T14:35:24.024 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.479534521000001, 2024-04-24T14:35:24.024 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044721860000000004, 2024-04-24T14:35:24.024 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:24.024 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:24.024 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:24.024 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5674", 2024-04-24T14:35:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/1498157813", 2024-04-24T14:35:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 200 2024-04-24T14:35:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470076+0000", 2024-04-24T14:35:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470076+0000", 2024-04-24T14:35:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:24.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470075+0000", 2024-04-24T14:35:24.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:24.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:24.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470078+0000", 2024-04-24T14:35:24.026 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.5459999999999998e-06 2024-04-24T14:35:24.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.026 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.026 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:24.026 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470083+0000", 2024-04-24T14:35:24.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 5.2009999999999998e-06 2024-04-24T14:35:24.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470088+0000", 2024-04-24T14:35:24.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4.7480000000000001e-06 2024-04-24T14:35:24.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:24.027 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473569+0000", 2024-04-24T14:35:24.027 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0034816399999999998 2024-04-24T14:35:24.027 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.027 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.027 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:24.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473887+0000", 2024-04-24T14:35:24.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00031747200000000001 2024-04-24T14:35:24.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.028 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:24.028 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.474548+0000", 2024-04-24T14:35:24.028 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00066107099999999995 2024-04-24T14:35:24.028 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.028 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:24.028 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.028 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.028 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "MOSDPGPush(2.2 108/107 [PushOp(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head, version: 49'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4ac2bf96:::rbd_data.12ff64619e2d.000000000000000f:head@49'3, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-24T14:35:24.029 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.470456+0000", 2024-04-24T14:35:24.029 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.479154487999999, 2024-04-24T14:35:24.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0055876789999999999, 2024-04-24T14:35:24.029 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:24.029 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "started", 2024-04-24T14:35:24.029 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:24.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:24.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470456+0000", 2024-04-24T14:35:24.029 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.029 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.029 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.029 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:24.029 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470456+0000", 2024-04-24T14:35:24.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:24.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470454+0000", 2024-04-24T14:35:24.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:24.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.030 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:24.030 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.470992+0000", 2024-04-24T14:35:24.030 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00053743599999999997 2024-04-24T14:35:24.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:24.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.471024+0000", 2024-04-24T14:35:24.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.2480000000000001e-05 2024-04-24T14:35:24.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.471045+0000", 2024-04-24T14:35:24.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0503e-05 2024-04-24T14:35:24.031 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.031 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.031 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:24.031 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473545+0000", 2024-04-24T14:35:24.031 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0025000299999999999 2024-04-24T14:35:24.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:24.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.473565+0000", 2024-04-24T14:35:24.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 2.0352000000000001e-05 2024-04-24T14:35:24.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.032 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.032 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:24.032 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.476043+0000", 2024-04-24T14:35:24.032 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0024783840000000001 2024-04-24T14:35:24.032 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.032 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:24.032 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.033 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.033 INFO:teuthology.orchestra.run.smithi103.stdout: "description": "osd_op(client.5606.0:3 3.2 3:5592b7cf:::journal.12ff64619e2d:head [watch watch cookie 94231437332288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-04-24T14:35:24.033 INFO:teuthology.orchestra.run.smithi103.stdout: "initiated_at": "2024-04-24T14:35:04.527849+0000", 2024-04-24T14:35:24.033 INFO:teuthology.orchestra.run.smithi103.stdout: "age": 19.421760930000001, 2024-04-24T14:35:24.033 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.0044396410000000002, 2024-04-24T14:35:24.033 INFO:teuthology.orchestra.run.smithi103.stdout: "type_data": { 2024-04-24T14:35:24.033 INFO:teuthology.orchestra.run.smithi103.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:24.033 INFO:teuthology.orchestra.run.smithi103.stdout: "client_info": { 2024-04-24T14:35:24.033 INFO:teuthology.orchestra.run.smithi103.stdout: "client": "client.5606", 2024-04-24T14:35:24.033 INFO:teuthology.orchestra.run.smithi103.stdout: "client_addr": "172.21.15.173:0/2386300588", 2024-04-24T14:35:24.033 INFO:teuthology.orchestra.run.smithi103.stdout: "tid": 3 2024-04-24T14:35:24.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.033 INFO:teuthology.orchestra.run.smithi103.stdout: "events": [ 2024-04-24T14:35:24.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "initiated", 2024-04-24T14:35:24.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527849+0000", 2024-04-24T14:35:24.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "throttled", 2024-04-24T14:35:24.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527849+0000", 2024-04-24T14:35:24.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0 2024-04-24T14:35:24.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.034 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.034 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "header_read", 2024-04-24T14:35:24.034 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527848+0000", 2024-04-24T14:35:24.034 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:24.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "all_read", 2024-04-24T14:35:24.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527855+0000", 2024-04-24T14:35:24.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 6.8719999999999996e-06 2024-04-24T14:35:24.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "dispatched", 2024-04-24T14:35:24.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527890+0000", 2024-04-24T14:35:24.035 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 3.4836999999999999e-05 2024-04-24T14:35:24.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.035 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.035 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.035 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.527906+0000", 2024-04-24T14:35:24.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6535999999999999e-05 2024-04-24T14:35:24.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "reached_pg", 2024-04-24T14:35:24.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.529465+0000", 2024-04-24T14:35:24.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001559429 2024-04-24T14:35:24.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "started", 2024-04-24T14:35:24.036 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.529800+0000", 2024-04-24T14:35:24.036 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.000334638 2024-04-24T14:35:24.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.036 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.036 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:24.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.530413+0000", 2024-04-24T14:35:24.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00061266800000000002 2024-04-24T14:35:24.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:24.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532044+0000", 2024-04-24T14:35:24.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.001631565 2024-04-24T14:35:24.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.037 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "op_commit", 2024-04-24T14:35:24.037 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532061+0000", 2024-04-24T14:35:24.037 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 1.6874000000000001e-05 2024-04-24T14:35:24.037 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.037 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "commit_sent", 2024-04-24T14:35:24.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532139+0000", 2024-04-24T14:35:24.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 7.8191999999999999e-05 2024-04-24T14:35:24.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-04-24T14:35:24.038 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-04-24T14:35:24.038 INFO:teuthology.orchestra.run.smithi103.stdout: "event": "done", 2024-04-24T14:35:24.038 INFO:teuthology.orchestra.run.smithi103.stdout: "time": "2024-04-24T14:35:04.532289+0000", 2024-04-24T14:35:24.038 INFO:teuthology.orchestra.run.smithi103.stdout: "duration": 0.00014948199999999999 2024-04-24T14:35:24.038 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.038 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:24.038 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.038 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-04-24T14:35:24.038 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-04-24T14:35:24.038 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-04-24T14:35:24.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-24T14:35:24.146 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2024-04-24T14:35:24.146 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:24.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-24T14:35:24.254 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:24.254 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2024-04-24T14:35:24.254 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2024-04-24T14:35:24.255 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2024-04-24T14:35:24.255 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:24.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-24T14:35:24.364 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2024-04-24T14:35:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2024-04-24T14:35:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2024-04-24T14:35:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:16 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 0~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.487655380999996, 2024-04-24T14:35:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.029643224, 2024-04-24T14:35:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 16 2024-04-24T14:35:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879835+0000", 2024-04-24T14:35:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879834+0000", 2024-04-24T14:35:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879840+0000", 2024-04-24T14:35:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4069999999999998e-06 2024-04-24T14:35:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879851+0000", 2024-04-24T14:35:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2024-04-24T14:35:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879857+0000", 2024-04-24T14:35:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1129999999999997e-06 2024-04-24T14:35:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881869+0000", 2024-04-24T14:35:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020125239999999999 2024-04-24T14:35:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.881880+0000", 2024-04-24T14:35:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1192000000000001e-05 2024-04-24T14:35:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879558+0000", 2024-04-24T14:35:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99767739200000005 2024-04-24T14:35:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879764+0000", 2024-04-24T14:35:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000206374 2024-04-24T14:35:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906738+0000", 2024-04-24T14:35:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.026974063 2024-04-24T14:35:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.906926+0000", 2024-04-24T14:35:24.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000187407 2024-04-24T14:35:24.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909479+0000", 2024-04-24T14:35:24.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025529960000000001 2024-04-24T14:35:24.370 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.370 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.370 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.371 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:17 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 512~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:24.371 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:24.371 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.487626395999996, 2024-04-24T14:35:24.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0301494529999999, 2024-04-24T14:35:24.371 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.371 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:24.371 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.371 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:24.371 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:24.371 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 17 2024-04-24T14:35:24.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.371 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:24.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:24.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879864+0000", 2024-04-24T14:35:24.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879867+0000", 2024-04-24T14:35:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.596e-06 2024-04-24T14:35:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879874+0000", 2024-04-24T14:35:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5729999999999996e-06 2024-04-24T14:35:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879878+0000", 2024-04-24T14:35:24.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.155e-06 2024-04-24T14:35:24.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882182+0000", 2024-04-24T14:35:24.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002303551 2024-04-24T14:35:24.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:24.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882190+0000", 2024-04-24T14:35:24.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1489999999999994e-06 2024-04-24T14:35:24.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.879956+0000", 2024-04-24T14:35:24.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997766402 2024-04-24T14:35:24.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:24.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880056+0000", 2024-04-24T14:35:24.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9735000000000006e-05 2024-04-24T14:35:24.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909554+0000", 2024-04-24T14:35:24.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029498021999999999 2024-04-24T14:35:24.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.909744+0000", 2024-04-24T14:35:24.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018987499999999999 2024-04-24T14:35:24.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910014+0000", 2024-04-24T14:35:24.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026994900000000001 2024-04-24T14:35:24.376 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.376 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.376 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:18 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1024~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.487607910999998, 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.030380909, 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:24.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879883+0000", 2024-04-24T14:35:24.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879882+0000", 2024-04-24T14:35:24.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:24.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879885+0000", 2024-04-24T14:35:24.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4169999999999999e-06 2024-04-24T14:35:24.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879891+0000", 2024-04-24T14:35:24.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0020000000000003e-06 2024-04-24T14:35:24.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879894+0000", 2024-04-24T14:35:24.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.219e-06 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882214+0000", 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023199890000000002 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882221+0000", 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4180000000000002e-06 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880124+0000", 2024-04-24T14:35:24.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.997903286 2024-04-24T14:35:24.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:24.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880211+0000", 2024-04-24T14:35:24.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7095999999999994e-05 2024-04-24T14:35:24.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910031+0000", 2024-04-24T14:35:24.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029819944000000001 2024-04-24T14:35:24.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910123+0000", 2024-04-24T14:35:24.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2229000000000001e-05 2024-04-24T14:35:24.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910264+0000", 2024-04-24T14:35:24.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014073300000000001 2024-04-24T14:35:24.382 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.382 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.382 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.382 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:19 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2048~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:24.382 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:24.383 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.487592629999995, 2024-04-24T14:35:24.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0308261089999999, 2024-04-24T14:35:24.383 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.383 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:24.383 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.383 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:24.383 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:24.383 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2024-04-24T14:35:24.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.383 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:24.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:24.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879898+0000", 2024-04-24T14:35:24.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:24.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879900+0000", 2024-04-24T14:35:24.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5859999999999999e-06 2024-04-24T14:35:24.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879906+0000", 2024-04-24T14:35:24.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9179999999999999e-06 2024-04-24T14:35:24.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879910+0000", 2024-04-24T14:35:24.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6550000000000002e-06 2024-04-24T14:35:24.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882242+0000", 2024-04-24T14:35:24.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023321309999999999 2024-04-24T14:35:24.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:24.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882248+0000", 2024-04-24T14:35:24.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2024-04-24T14:35:24.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880259+0000", 2024-04-24T14:35:24.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99801182799999999 2024-04-24T14:35:24.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:24.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880339+0000", 2024-04-24T14:35:24.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9463999999999997e-05 2024-04-24T14:35:24.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910328+0000", 2024-04-24T14:35:24.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029989202 2024-04-24T14:35:24.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910458+0000", 2024-04-24T14:35:24.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129543 2024-04-24T14:35:24.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910724+0000", 2024-04-24T14:35:24.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026665899999999998 2024-04-24T14:35:24.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.388 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.388 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:20 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 1536~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:24.388 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:24.388 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.487576849000007, 2024-04-24T14:35:24.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0310641039999999, 2024-04-24T14:35:24.389 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.389 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:24.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:24.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:24.389 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 20 2024-04-24T14:35:24.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.389 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:24.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:24.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879914+0000", 2024-04-24T14:35:24.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:24.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879916+0000", 2024-04-24T14:35:24.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.12e-06 2024-04-24T14:35:24.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879921+0000", 2024-04-24T14:35:24.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2599999999999996e-06 2024-04-24T14:35:24.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879924+0000", 2024-04-24T14:35:24.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.241e-06 2024-04-24T14:35:24.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882267+0000", 2024-04-24T14:35:24.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023426369999999998 2024-04-24T14:35:24.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:24.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882273+0000", 2024-04-24T14:35:24.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2770000000000002e-06 2024-04-24T14:35:24.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880388+0000", 2024-04-24T14:35:24.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99811475000000005 2024-04-24T14:35:24.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:24.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880501+0000", 2024-04-24T14:35:24.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113429 2024-04-24T14:35:24.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910734+0000", 2024-04-24T14:35:24.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030232774 2024-04-24T14:35:24.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910817+0000", 2024-04-24T14:35:24.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2887999999999997e-05 2024-04-24T14:35:24.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.910978+0000", 2024-04-24T14:35:24.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016113400000000001 2024-04-24T14:35:24.394 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.394 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.394 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.394 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:21 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 2560~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:24.394 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:24.394 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.487562561000004, 2024-04-24T14:35:24.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031667444, 2024-04-24T14:35:24.395 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.395 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:24.395 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.395 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:24.395 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:24.395 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 21 2024-04-24T14:35:24.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.395 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:24.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:24.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879928+0000", 2024-04-24T14:35:24.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:24.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879930+0000", 2024-04-24T14:35:24.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.108e-06 2024-04-24T14:35:24.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879938+0000", 2024-04-24T14:35:24.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5349999999999999e-06 2024-04-24T14:35:24.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879943+0000", 2024-04-24T14:35:24.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1649999999999997e-06 2024-04-24T14:35:24.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882292+0000", 2024-04-24T14:35:24.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002349571 2024-04-24T14:35:24.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:24.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882298+0000", 2024-04-24T14:35:24.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9959999999999999e-06 2024-04-24T14:35:24.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880570+0000", 2024-04-24T14:35:24.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99827173800000002 2024-04-24T14:35:24.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:24.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880640+0000", 2024-04-24T14:35:24.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9564e-05 2024-04-24T14:35:24.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911047+0000", 2024-04-24T14:35:24.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030407533 2024-04-24T14:35:24.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911253+0000", 2024-04-24T14:35:24.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020568499999999999 2024-04-24T14:35:24.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911596+0000", 2024-04-24T14:35:24.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034288999999999999 2024-04-24T14:35:24.400 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.400 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.400 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.400 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:22 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3072~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:24.400 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:24.400 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.487542809000004, 2024-04-24T14:35:24.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.031882422, 2024-04-24T14:35:24.400 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.400 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:24.401 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.401 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:24.401 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:24.401 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 22 2024-04-24T14:35:24.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.401 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:24.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:24.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879948+0000", 2024-04-24T14:35:24.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:24.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879950+0000", 2024-04-24T14:35:24.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2709999999999999e-06 2024-04-24T14:35:24.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879957+0000", 2024-04-24T14:35:24.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9260000000000003e-06 2024-04-24T14:35:24.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879961+0000", 2024-04-24T14:35:24.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5870000000000003e-06 2024-04-24T14:35:24.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882318+0000", 2024-04-24T14:35:24.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023570100000000001 2024-04-24T14:35:24.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:24.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882326+0000", 2024-04-24T14:35:24.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8669999999999994e-06 2024-04-24T14:35:24.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880719+0000", 2024-04-24T14:35:24.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99839267700000001 2024-04-24T14:35:24.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:24.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880816+0000", 2024-04-24T14:35:24.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7367999999999993e-05 2024-04-24T14:35:24.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911617+0000", 2024-04-24T14:35:24.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030800306999999999 2024-04-24T14:35:24.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911694+0000", 2024-04-24T14:35:24.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6903000000000006e-05 2024-04-24T14:35:24.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911830+0000", 2024-04-24T14:35:24.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013689400000000001 2024-04-24T14:35:24.406 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.406 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.406 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.406 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5215.0:23 3.2 3:5ad2259e:ns::rbd_data.1294a8af6a1d.0000000000000000:head [read 3584~512] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e67)", 2024-04-24T14:35:24.406 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:24.406 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.487525362, 2024-04-24T14:35:24.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0324686009999999, 2024-04-24T14:35:24.406 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.406 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:24.406 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.407 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5215", 2024-04-24T14:35:24.407 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/1209186671", 2024-04-24T14:35:24.407 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 23 2024-04-24T14:35:24.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.407 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:24.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:24.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879965+0000", 2024-04-24T14:35:24.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:24.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879973+0000", 2024-04-24T14:35:24.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4619999999999999e-06 2024-04-24T14:35:24.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879978+0000", 2024-04-24T14:35:24.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6969999999999998e-06 2024-04-24T14:35:24.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.879982+0000", 2024-04-24T14:35:24.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4539999999999999e-06 2024-04-24T14:35:24.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882348+0000", 2024-04-24T14:35:24.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023661289999999998 2024-04-24T14:35:24.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2024-04-24T14:35:24.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:01.882354+0000", 2024-04-24T14:35:24.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2140000000000001e-06 2024-04-24T14:35:24.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880883+0000", 2024-04-24T14:35:24.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.998529467 2024-04-24T14:35:24.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for missing object", 2024-04-24T14:35:24.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.880950+0000", 2024-04-24T14:35:24.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6260999999999995e-05 2024-04-24T14:35:24.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.911901+0000", 2024-04-24T14:35:24.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030951698 2024-04-24T14:35:24.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912091+0000", 2024-04-24T14:35:24.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019000000000000001 2024-04-24T14:35:24.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:02.912434+0000", 2024-04-24T14:35:24.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034262399999999998 2024-04-24T14:35:24.412 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.412 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.412 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.412 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:100 3.2 3:7cbd3bdf:::rbd_data.12ff64619e2d.0000000000000002:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:24.412 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:24.412 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.613470546999999, 2024-04-24T14:35:24.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.009078015, 2024-04-24T14:35:24.412 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.412 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:24.412 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.412 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:24.413 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:24.413 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 100 2024-04-24T14:35:24.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.413 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:24.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754020+0000", 2024-04-24T14:35:24.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754019+0000", 2024-04-24T14:35:24.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:24.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754025+0000", 2024-04-24T14:35:24.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2110000000000003e-06 2024-04-24T14:35:24.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754037+0000", 2024-04-24T14:35:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1868999999999999e-05 2024-04-24T14:35:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754046+0000", 2024-04-24T14:35:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7430000000000008e-06 2024-04-24T14:35:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754095+0000", 2024-04-24T14:35:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8924000000000001e-05 2024-04-24T14:35:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754153+0000", 2024-04-24T14:35:24.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8165000000000003e-05 2024-04-24T14:35:24.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:24.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.754262+0000", 2024-04-24T14:35:24.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108694 2024-04-24T14:35:24.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:24.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.762999+0000", 2024-04-24T14:35:24.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0087372590000000007 2024-04-24T14:35:24.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763027+0000", 2024-04-24T14:35:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8467e-05 2024-04-24T14:35:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763070+0000", 2024-04-24T14:35:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2236e-05 2024-04-24T14:35:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.763098+0000", 2024-04-24T14:35:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8561999999999999e-05 2024-04-24T14:35:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:101 3.0 3:27c6cb83:::rbd_data.12ff64619e2d.0000000000000003:head [delete] snapc 7=[7,6,4] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.408262880999999, 2024-04-24T14:35:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012894743, 2024-04-24T14:35:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 101 2024-04-24T14:35:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959228+0000", 2024-04-24T14:35:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959227+0000", 2024-04-24T14:35:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959233+0000", 2024-04-24T14:35:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.857e-06 2024-04-24T14:35:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959249+0000", 2024-04-24T14:35:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6155999999999999e-05 2024-04-24T14:35:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959257+0000", 2024-04-24T14:35:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7440000000000007e-06 2024-04-24T14:35:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959790+0000", 2024-04-24T14:35:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053381100000000003 2024-04-24T14:35:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960206+0000", 2024-04-24T14:35:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041549700000000002 2024-04-24T14:35:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.960744+0000", 2024-04-24T14:35:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053762199999999997 2024-04-24T14:35:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971963+0000", 2024-04-24T14:35:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011219816000000001 2024-04-24T14:35:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972021+0000", 2024-04-24T14:35:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7855000000000001e-05 2024-04-24T14:35:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972073+0000", 2024-04-24T14:35:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2196e-05 2024-04-24T14:35:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972123+0000", 2024-04-24T14:35:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9311999999999998e-05 2024-04-24T14:35:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:102 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.408222854999998, 2024-04-24T14:35:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012939713, 2024-04-24T14:35:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 102 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959268+0000", 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959267+0000", 2024-04-24T14:35:24.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:24.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959274+0000", 2024-04-24T14:35:24.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.593e-06 2024-04-24T14:35:24.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959291+0000", 2024-04-24T14:35:24.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6387000000000001e-05 2024-04-24T14:35:24.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959301+0000", 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0023e-05 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969594+0000", 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010293182 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.969902+0000", 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030835299999999998 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970408+0000", 2024-04-24T14:35:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050588100000000002 2024-04-24T14:35:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972009+0000", 2024-04-24T14:35:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016011210000000001 2024-04-24T14:35:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972132+0000", 2024-04-24T14:35:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122521 2024-04-24T14:35:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972174+0000", 2024-04-24T14:35:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2242e-05 2024-04-24T14:35:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972208+0000", 2024-04-24T14:35:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3522999999999997e-05 2024-04-24T14:35:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:104 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.407611641000003, 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014589062, 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 104 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959879+0000", 2024-04-24T14:35:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959877+0000", 2024-04-24T14:35:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2024-04-24T14:35:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959890+0000", 2024-04-24T14:35:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2537e-05 2024-04-24T14:35:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959909+0000", 2024-04-24T14:35:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9340000000000001e-05 2024-04-24T14:35:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959922+0000", 2024-04-24T14:35:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2882e-05 2024-04-24T14:35:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.970900+0000", 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010977736 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971039+0000", 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139593 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.971453+0000", 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041351099999999998 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:24.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974044+0000", 2024-04-24T14:35:24.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025908289999999998 2024-04-24T14:35:24.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:24.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974377+0000", 2024-04-24T14:35:24.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033310399999999997 2024-04-24T14:35:24.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:24.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974433+0000", 2024-04-24T14:35:24.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6166999999999999e-05 2024-04-24T14:35:24.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.974468+0000", 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5336000000000002e-05 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:106 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.407552308, 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017192064999999999, 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.435 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 106 2024-04-24T14:35:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959939+0000", 2024-04-24T14:35:24.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959938+0000", 2024-04-24T14:35:24.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:24.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959943+0000", 2024-04-24T14:35:24.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.677e-06 2024-04-24T14:35:24.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959953+0000", 2024-04-24T14:35:24.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8069999999999994e-06 2024-04-24T14:35:24.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.959959+0000", 2024-04-24T14:35:24.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0560000000000001e-06 2024-04-24T14:35:24.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972232+0000", 2024-04-24T14:35:24.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012272460000000001 2024-04-24T14:35:24.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972304+0000", 2024-04-24T14:35:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2630000000000001e-05 2024-04-24T14:35:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.972548+0000", 2024-04-24T14:35:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000244047 2024-04-24T14:35:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.976900+0000", 2024-04-24T14:35:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043514069999999998 2024-04-24T14:35:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977076+0000", 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176281 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977106+0000", 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0015e-05 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.977131+0000", 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4669e-05 2024-04-24T14:35:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:114 3.2 3:4eaf8166:::rbd_data.12ff64619e2d.0000000000000008:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.384231143999997, 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0090880089999999993, 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 114 2024-04-24T14:35:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983260+0000", 2024-04-24T14:35:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983259+0000", 2024-04-24T14:35:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983264+0000", 2024-04-24T14:35:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9130000000000002e-06 2024-04-24T14:35:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983277+0000", 2024-04-24T14:35:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3723999999999999e-05 2024-04-24T14:35:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983282+0000", 2024-04-24T14:35:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0429999999999998e-06 2024-04-24T14:35:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983339+0000", 2024-04-24T14:35:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6323000000000003e-05 2024-04-24T14:35:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983464+0000", 2024-04-24T14:35:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012554299999999999 2024-04-24T14:35:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:24.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.983693+0000", 2024-04-24T14:35:24.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000228848 2024-04-24T14:35:24.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:24.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.986978+0000", 2024-04-24T14:35:24.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032852810000000001 2024-04-24T14:35:24.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:24.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992288+0000", 2024-04-24T14:35:24.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0053095369999999996 2024-04-24T14:35:24.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:24.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992322+0000", 2024-04-24T14:35:24.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3683999999999998e-05 2024-04-24T14:35:24.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.992348+0000", 2024-04-24T14:35:24.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6157999999999999e-05 2024-04-24T14:35:24.446 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.446 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.446 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.447 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:121 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:24.447 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:24.447 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.372175044999999, 2024-04-24T14:35:24.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0098798159999999996, 2024-04-24T14:35:24.447 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.447 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:24.447 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.447 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:24.447 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:24.448 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 121 2024-04-24T14:35:24.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.448 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:24.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995316+0000", 2024-04-24T14:35:24.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995315+0000", 2024-04-24T14:35:24.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:24.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995320+0000", 2024-04-24T14:35:24.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0239999999999997e-06 2024-04-24T14:35:24.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995328+0000", 2024-04-24T14:35:24.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7800000000000001e-06 2024-04-24T14:35:24.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.995334+0000", 2024-04-24T14:35:24.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2330000000000003e-06 2024-04-24T14:35:24.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998810+0000", 2024-04-24T14:35:24.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034756090000000002 2024-04-24T14:35:24.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.998870+0000", 2024-04-24T14:35:24.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0625999999999999e-05 2024-04-24T14:35:24.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:24.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:26.999091+0000", 2024-04-24T14:35:24.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000220631 2024-04-24T14:35:24.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:24.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.001389+0000", 2024-04-24T14:35:24.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022977549999999998 2024-04-24T14:35:24.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:24.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005097+0000", 2024-04-24T14:35:24.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0037085439999999998 2024-04-24T14:35:24.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:24.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005154+0000", 2024-04-24T14:35:24.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6208000000000001e-05 2024-04-24T14:35:24.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005196+0000", 2024-04-24T14:35:24.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2052000000000003e-05 2024-04-24T14:35:24.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.453 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.453 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:127 3.0 3:26292d44:::rbd_object_map.12ff64619e2d:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:24.453 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:24.453 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.361873529, 2024-04-24T14:35:24.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.013414918, 2024-04-24T14:35:24.453 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.453 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:24.453 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.453 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2024-04-24T14:35:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005617+0000", 2024-04-24T14:35:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005616+0000", 2024-04-24T14:35:24.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:24.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005622+0000", 2024-04-24T14:35:24.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2020000000000003e-06 2024-04-24T14:35:24.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005632+0000", 2024-04-24T14:35:24.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2024-04-24T14:35:24.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005637+0000", 2024-04-24T14:35:24.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7439999999999998e-06 2024-04-24T14:35:24.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005947+0000", 2024-04-24T14:35:24.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000310792 2024-04-24T14:35:24.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006059+0000", 2024-04-24T14:35:24.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011192399999999999 2024-04-24T14:35:24.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:24.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006503+0000", 2024-04-24T14:35:24.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000443394 2024-04-24T14:35:24.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:24.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018864+0000", 2024-04-24T14:35:24.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.012361162 2024-04-24T14:35:24.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:24.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018877+0000", 2024-04-24T14:35:24.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3497e-05 2024-04-24T14:35:24.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:24.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018965+0000", 2024-04-24T14:35:24.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7476000000000001e-05 2024-04-24T14:35:24.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019032+0000", 2024-04-24T14:35:24.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7469000000000001e-05 2024-04-24T14:35:24.459 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.459 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.459 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.459 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:129 3.2 3:40847be4:::rbd_data.12ff64619e2d.000000000000000b:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:24.459 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:24.459 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.361733295999997, 2024-04-24T14:35:24.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010059754000000001, 2024-04-24T14:35:24.459 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.459 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:24.459 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.459 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:24.459 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 129 2024-04-24T14:35:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005758+0000", 2024-04-24T14:35:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005757+0000", 2024-04-24T14:35:24.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2024-04-24T14:35:24.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005760+0000", 2024-04-24T14:35:24.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2430000000000001e-06 2024-04-24T14:35:24.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005769+0000", 2024-04-24T14:35:24.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4810000000000006e-06 2024-04-24T14:35:24.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005773+0000", 2024-04-24T14:35:24.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9550000000000002e-06 2024-04-24T14:35:24.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005840+0000", 2024-04-24T14:35:24.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6934999999999995e-05 2024-04-24T14:35:24.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.005961+0000", 2024-04-24T14:35:24.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012153 2024-04-24T14:35:24.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3", 2024-04-24T14:35:24.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006181+0000", 2024-04-24T14:35:24.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021947500000000001 2024-04-24T14:35:24.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:24.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015520+0000", 2024-04-24T14:35:24.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093393049999999991 2024-04-24T14:35:24.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:24.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015733+0000", 2024-04-24T14:35:24.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021355699999999999 2024-04-24T14:35:24.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:24.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015773+0000", 2024-04-24T14:35:24.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9554000000000001e-05 2024-04-24T14:35:24.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015817+0000", 2024-04-24T14:35:24.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4325e-05 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:131 3.0 3:2a98ea1c:::rbd_data.12ff64619e2d.000000000000000d:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.361455415000002, 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014251976, 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:24.465 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 131 2024-04-24T14:35:24.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.466 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:24.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006035+0000", 2024-04-24T14:35:24.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006034+0000", 2024-04-24T14:35:24.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2024-04-24T14:35:24.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006042+0000", 2024-04-24T14:35:24.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2970000000000004e-06 2024-04-24T14:35:24.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006061+0000", 2024-04-24T14:35:24.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8505999999999999e-05 2024-04-24T14:35:24.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006071+0000", 2024-04-24T14:35:24.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0409e-05 2024-04-24T14:35:24.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007899+0000", 2024-04-24T14:35:24.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018274789999999999 2024-04-24T14:35:24.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008281+0000", 2024-04-24T14:35:24.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038244700000000001 2024-04-24T14:35:24.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:24.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.008528+0000", 2024-04-24T14:35:24.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024723900000000001 2024-04-24T14:35:24.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:24.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019070+0000", 2024-04-24T14:35:24.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010541867999999999 2024-04-24T14:35:24.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:24.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020099+0000", 2024-04-24T14:35:24.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00102852 2024-04-24T14:35:24.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:24.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020226+0000", 2024-04-24T14:35:24.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012744300000000001 2024-04-24T14:35:24.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020287+0000", 2024-04-24T14:35:24.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1253000000000005e-05 2024-04-24T14:35:24.470 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.470 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.470 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.471 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.5324.0:132 3.0 3:350dfac4:::rbd_data.12ff64619e2d.000000000000000e:head [delete] snapc 7=[7,6,4] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-24T14:35:24.471 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:24.471 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.361401561999998, 2024-04-24T14:35:24.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015917981000000001, 2024-04-24T14:35:24.471 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.471 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-24T14:35:24.471 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2024-04-24T14:35:24.471 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.5324", 2024-04-24T14:35:24.471 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.173:0/3159276725", 2024-04-24T14:35:24.471 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 132 2024-04-24T14:35:24.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.472 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:24.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:24.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006089+0000", 2024-04-24T14:35:24.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:24.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006093+0000", 2024-04-24T14:35:24.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3819999999999997e-06 2024-04-24T14:35:24.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006103+0000", 2024-04-24T14:35:24.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0352e-05 2024-04-24T14:35:24.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.006110+0000", 2024-04-24T14:35:24.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3910000000000003e-06 2024-04-24T14:35:24.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015586+0000", 2024-04-24T14:35:24.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094759119999999995 2024-04-24T14:35:24.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015730+0000", 2024-04-24T14:35:24.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014487199999999999 2024-04-24T14:35:24.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2", 2024-04-24T14:35:24.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.015886+0000", 2024-04-24T14:35:24.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015573799999999999 2024-04-24T14:35:24.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2024-04-24T14:35:24.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.020306+0000", 2024-04-24T14:35:24.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0044199399999999998 2024-04-24T14:35:24.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2024-04-24T14:35:24.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021887+0000", 2024-04-24T14:35:24.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015809400000000001 2024-04-24T14:35:24.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2024-04-24T14:35:24.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.021942+0000", 2024-04-24T14:35:24.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5071000000000002e-05 2024-04-24T14:35:24.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.022007+0000", 2024-04-24T14:35:24.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5110000000000005e-05 2024-04-24T14:35:24.476 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.476 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.476 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.5324.0:127 3.0 e87/69 ondisk, result = 0)", 2024-04-24T14:35:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.359692494000001, 2024-04-24T14:35:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011245125, 2024-04-24T14:35:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2024-04-24T14:35:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2024-04-24T14:35:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2024-04-24T14:35:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2024-04-24T14:35:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2024-04-24T14:35:24.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:24.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2024-04-24T14:35:24.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2024-04-24T14:35:24.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007798+0000", 2024-04-24T14:35:24.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2024-04-24T14:35:24.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2024-04-24T14:35:24.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007804+0000", 2024-04-24T14:35:24.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6610000000000003e-06 2024-04-24T14:35:24.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2024-04-24T14:35:24.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007817+0000", 2024-04-24T14:35:24.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3030999999999999e-05 2024-04-24T14:35:24.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2024-04-24T14:35:24.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.007832+0000", 2024-04-24T14:35:24.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.47e-05 2024-04-24T14:35:24.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2024-04-24T14:35:24.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018843+0000", 2024-04-24T14:35:24.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.011010647 2024-04-24T14:35:24.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2024-04-24T14:35:24.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.018853+0000", 2024-04-24T14:35:24.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319000000000001e-05 2024-04-24T14:35:24.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-04-24T14:35:24.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-04-24T14:35:24.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2024-04-24T14:35:24.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2024-04-24T14:34:27.019043+0000", 2024-04-24T14:35:24.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019055600000000001 2024-04-24T14:35:24.480 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.481 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.481 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.481 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-04-24T14:35:24.481 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-04-24T14:35:24.481 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:24.481 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-24T14:35:24.848 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2024-04-24T14:35:24.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-24T14:35:25.826 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-24T14:35:25.827 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-04-24T14:35:25.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-04-24T14:35:25.863 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2024-04-24T14:35:26.129 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-04-24T14:35:26.130+0000 7ff9c25de700 -1 mon.a@0(leader).osd e124 definitely_dead 0 2024-04-24T14:35:26.869 INFO:teuthology.orchestra.run.smithi090.stderr:marked in osd.1. 2024-04-24T14:35:26.880 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-04-24T14:35:26.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-24T14:35:27.005 INFO:teuthology.orchestra.run.smithi090.stdout:mon.a: {} 2024-04-24T14:35:27.026 INFO:teuthology.orchestra.run.smithi090.stdout:mon.b: {} 2024-04-24T14:35:27.037 INFO:teuthology.orchestra.run.smithi090.stdout:mon.c: {} 2024-04-24T14:35:27.039 INFO:teuthology.orchestra.run.smithi090.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T14:35:27.040 INFO:teuthology.orchestra.run.smithi090.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T14:35:27.040 INFO:teuthology.orchestra.run.smithi090.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-24T14:35:27.052 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-24T14:35:27.880 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.0 to 1 (10000) 2024-04-24T14:35:27.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-24T14:35:28.881 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-24T14:35:28.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-24T14:35:29.903 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.2 to 1 (10000) 2024-04-24T14:35:29.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-24T14:35:30.905 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-24T14:35:30.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-24T14:35:31.914 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.3 to 1 (10000) 2024-04-24T14:35:31.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-04-24T14:35:32.924 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-24T14:35:32.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-24T14:35:33.938 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.4 to 1 (10000) 2024-04-24T14:35:33.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-04-24T14:35:34.945 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-24T14:35:34.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-24T14:35:35.954 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.5 to 1 (10000) 2024-04-24T14:35:35.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-04-24T14:35:36.964 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-24T14:35:36.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-24T14:35:37.970 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.1 to 1 (10000) 2024-04-24T14:35:37.984 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-24T14:35:38.984 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-24T14:35:38.996 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-24T14:35:38.996 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-24T14:35:38.996 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-24T14:35:38.996 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-24T14:35:38.996 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-24T14:35:38.997 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T14:35:39.287 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:39.287 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":137,"fsid":"6706935e-d09a-472b-b839-321319fa9e54","created":"2024-04-24T14:32:09.059925+0000","modified":"2024-04-24T14:35:38.975191+0000","last_up_change":"2024-04-24T14:34:00.856741+0000","last_in_change":"2024-04-24T14:35:26.859498+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T14:32:21.233022+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T14:32:26.609964+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"21","last_force_op_resend_prenautilus":"21","last_force_op_resend_preluminous":"21","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":119,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-04-24T14:32:33.987564+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"21","last_force_op_resend_prenautilus":"21","last_force_op_resend_preluminous":"21","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":119,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":2,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a2506b4f-3486-4f51-972f-1d9e4b015346","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":20309},{"type":"v1","addr":"172.21.15.90:6817","nonce":20309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":20309},{"type":"v1","addr":"172.21.15.90:6819","nonce":20309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":20309},{"type":"v1","addr":"172.21.15.90:6823","nonce":20309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":20309},{"type":"v1","addr":"172.21.15.90:6821","nonce":20309}]},"public_addr":"172.21.15.90:6817/20309","cluster_addr":"172.21.15.90:6819/20309","heartbeat_back_addr":"172.21.15.90:6823/20309","heartbeat_front_addr":"172.21.15.90:6821/20309","state":["exists","up"]},{"osd":1,"uuid":"9e17324f-a615-494f-ba0d-7c8aa8dac95c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":31,"up_from":66,"up_thru":125,"down_at":32,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":34413},{"type":"v1","addr":"172.21.15.90:6801","nonce":34413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":34413},{"type":"v1","addr":"172.21.15.90:6803","nonce":34413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":34413},{"type":"v1","addr":"172.21.15.90:6807","nonce":34413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":34413},{"type":"v1","addr":"172.21.15.90:6805","nonce":34413}]},"public_addr":"172.21.15.90:6801/34413","cluster_addr":"172.21.15.90:6803/34413","heartbeat_back_addr":"172.21.15.90:6807/34413","heartbeat_front_addr":"172.21.15.90:6805/34413","state":["exists","up"]},{"osd":2,"uuid":"f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":20310},{"type":"v1","addr":"172.21.15.90:6809","nonce":20310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":20310},{"type":"v1","addr":"172.21.15.90:6811","nonce":20310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":20310},{"type":"v1","addr":"172.21.15.90:6815","nonce":20310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":20310},{"type":"v1","addr":"172.21.15.90:6813","nonce":20310}]},"public_addr":"172.21.15.90:6809/20310","cluster_addr":"172.21.15.90:6811/20310","heartbeat_back_addr":"172.21.15.90:6815/20310","heartbeat_front_addr":"172.21.15.90:6813/20310","state":["exists","up"]},{"osd":3,"uuid":"509c16da-22b2-48e8-aa51-c320c8c4fd8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":20316},{"type":"v1","addr":"172.21.15.103:6801","nonce":20316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":20316},{"type":"v1","addr":"172.21.15.103:6803","nonce":20316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":20316},{"type":"v1","addr":"172.21.15.103:6807","nonce":20316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":20316},{"type":"v1","addr":"172.21.15.103:6805","nonce":20316}]},"public_addr":"172.21.15.103:6801/20316","cluster_addr":"172.21.15.103:6803/20316","heartbeat_back_addr":"172.21.15.103:6807/20316","heartbeat_front_addr":"172.21.15.103:6805/20316","state":["exists","up"]},{"osd":4,"uuid":"1462066b-4e59-46d4-9b85-1f61ee1d56a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":20318},{"type":"v1","addr":"172.21.15.103:6809","nonce":20318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":20318},{"type":"v1","addr":"172.21.15.103:6811","nonce":20318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":20318},{"type":"v1","addr":"172.21.15.103:6815","nonce":20318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":20318},{"type":"v1","addr":"172.21.15.103:6813","nonce":20318}]},"public_addr":"172.21.15.103:6809/20318","cluster_addr":"172.21.15.103:6811/20318","heartbeat_back_addr":"172.21.15.103:6815/20318","heartbeat_front_addr":"172.21.15.103:6813/20318","state":["exists","up"]},{"osd":5,"uuid":"4dc954a8-5f3c-4826-b9bc-98cf30519382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":20317},{"type":"v1","addr":"172.21.15.103:6817","nonce":20317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":20317},{"type":"v1","addr":"172.21.15.103:6819","nonce":20317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":20317},{"type":"v1","addr":"172.21.15.103:6823","nonce":20317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":20317},{"type":"v1","addr":"172.21.15.103:6821","nonce":20317}]},"public_addr":"172.21.15.103:6817/20317","cluster_addr":"172.21.15.103:6819/20317","heartbeat_back_addr":"172.21.15.103:6823/20317","heartbeat_front_addr":"172.21.15.103:6821/20317","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.792899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.580414+0000","dead_epoch":32},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.713336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.579638+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.570510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.736558+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T14:35:39.301 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-24T14:35:39.301 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T14:35:39.588 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:39.588 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":137,"fsid":"6706935e-d09a-472b-b839-321319fa9e54","created":"2024-04-24T14:32:09.059925+0000","modified":"2024-04-24T14:35:38.975191+0000","last_up_change":"2024-04-24T14:34:00.856741+0000","last_in_change":"2024-04-24T14:35:26.859498+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T14:32:21.233022+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T14:32:26.609964+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"21","last_force_op_resend_prenautilus":"21","last_force_op_resend_preluminous":"21","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":119,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-04-24T14:32:33.987564+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"21","last_force_op_resend_prenautilus":"21","last_force_op_resend_preluminous":"21","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":119,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":2,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a2506b4f-3486-4f51-972f-1d9e4b015346","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":20309},{"type":"v1","addr":"172.21.15.90:6817","nonce":20309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":20309},{"type":"v1","addr":"172.21.15.90:6819","nonce":20309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":20309},{"type":"v1","addr":"172.21.15.90:6823","nonce":20309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":20309},{"type":"v1","addr":"172.21.15.90:6821","nonce":20309}]},"public_addr":"172.21.15.90:6817/20309","cluster_addr":"172.21.15.90:6819/20309","heartbeat_back_addr":"172.21.15.90:6823/20309","heartbeat_front_addr":"172.21.15.90:6821/20309","state":["exists","up"]},{"osd":1,"uuid":"9e17324f-a615-494f-ba0d-7c8aa8dac95c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":31,"up_from":66,"up_thru":125,"down_at":32,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":34413},{"type":"v1","addr":"172.21.15.90:6801","nonce":34413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":34413},{"type":"v1","addr":"172.21.15.90:6803","nonce":34413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":34413},{"type":"v1","addr":"172.21.15.90:6807","nonce":34413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":34413},{"type":"v1","addr":"172.21.15.90:6805","nonce":34413}]},"public_addr":"172.21.15.90:6801/34413","cluster_addr":"172.21.15.90:6803/34413","heartbeat_back_addr":"172.21.15.90:6807/34413","heartbeat_front_addr":"172.21.15.90:6805/34413","state":["exists","up"]},{"osd":2,"uuid":"f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":20310},{"type":"v1","addr":"172.21.15.90:6809","nonce":20310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":20310},{"type":"v1","addr":"172.21.15.90:6811","nonce":20310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":20310},{"type":"v1","addr":"172.21.15.90:6815","nonce":20310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":20310},{"type":"v1","addr":"172.21.15.90:6813","nonce":20310}]},"public_addr":"172.21.15.90:6809/20310","cluster_addr":"172.21.15.90:6811/20310","heartbeat_back_addr":"172.21.15.90:6815/20310","heartbeat_front_addr":"172.21.15.90:6813/20310","state":["exists","up"]},{"osd":3,"uuid":"509c16da-22b2-48e8-aa51-c320c8c4fd8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":20316},{"type":"v1","addr":"172.21.15.103:6801","nonce":20316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":20316},{"type":"v1","addr":"172.21.15.103:6803","nonce":20316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":20316},{"type":"v1","addr":"172.21.15.103:6807","nonce":20316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":20316},{"type":"v1","addr":"172.21.15.103:6805","nonce":20316}]},"public_addr":"172.21.15.103:6801/20316","cluster_addr":"172.21.15.103:6803/20316","heartbeat_back_addr":"172.21.15.103:6807/20316","heartbeat_front_addr":"172.21.15.103:6805/20316","state":["exists","up"]},{"osd":4,"uuid":"1462066b-4e59-46d4-9b85-1f61ee1d56a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":20318},{"type":"v1","addr":"172.21.15.103:6809","nonce":20318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":20318},{"type":"v1","addr":"172.21.15.103:6811","nonce":20318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":20318},{"type":"v1","addr":"172.21.15.103:6815","nonce":20318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":20318},{"type":"v1","addr":"172.21.15.103:6813","nonce":20318}]},"public_addr":"172.21.15.103:6809/20318","cluster_addr":"172.21.15.103:6811/20318","heartbeat_back_addr":"172.21.15.103:6815/20318","heartbeat_front_addr":"172.21.15.103:6813/20318","state":["exists","up"]},{"osd":5,"uuid":"4dc954a8-5f3c-4826-b9bc-98cf30519382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":20317},{"type":"v1","addr":"172.21.15.103:6817","nonce":20317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":20317},{"type":"v1","addr":"172.21.15.103:6819","nonce":20317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":20317},{"type":"v1","addr":"172.21.15.103:6823","nonce":20317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":20317},{"type":"v1","addr":"172.21.15.103:6821","nonce":20317}]},"public_addr":"172.21.15.103:6817/20317","cluster_addr":"172.21.15.103:6819/20317","heartbeat_back_addr":"172.21.15.103:6823/20317","heartbeat_front_addr":"172.21.15.103:6821/20317","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.792899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.580414+0000","dead_epoch":32},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.713336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.579638+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.570510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.736558+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T14:35:39.602 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-24T14:35:39.602 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-24T14:35:39.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-24T14:35:39.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-24T14:35:39.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-24T14:35:39.604 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-24T14:35:39.761 INFO:teuthology.orchestra.run.smithi090.stdout:47244640299 2024-04-24T14:35:39.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T14:35:39.764 INFO:teuthology.orchestra.run.smithi090.stdout:47244640301 2024-04-24T14:35:39.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T14:35:39.780 INFO:teuthology.orchestra.run.smithi090.stdout:47244640300 2024-04-24T14:35:39.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T14:35:39.787 INFO:teuthology.orchestra.run.smithi090.stdout:47244640301 2024-04-24T14:35:39.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T14:35:39.788 INFO:teuthology.orchestra.run.smithi090.stdout:47244640301 2024-04-24T14:35:39.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T14:35:39.798 INFO:teuthology.orchestra.run.smithi090.stdout:283467841557 2024-04-24T14:35:39.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T14:35:40.106 INFO:teuthology.orchestra.run.smithi090.stdout:47244640298 2024-04-24T14:35:40.119 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640299 got 47244640298 for osd.3 2024-04-24T14:35:40.122 INFO:teuthology.orchestra.run.smithi090.stdout:47244640299 2024-04-24T14:35:40.135 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640300 got 47244640299 for osd.0 2024-04-24T14:35:40.171 INFO:teuthology.orchestra.run.smithi090.stdout:47244640300 2024-04-24T14:35:40.185 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640301 got 47244640300 for osd.5 2024-04-24T14:35:40.232 INFO:teuthology.orchestra.run.smithi090.stdout:283467841556 2024-04-24T14:35:40.245 INFO:tasks.ceph.ceph_manager.ceph:need seq 283467841557 got 283467841556 for osd.1 2024-04-24T14:35:40.252 INFO:teuthology.orchestra.run.smithi090.stdout:47244640300 2024-04-24T14:35:40.258 INFO:teuthology.orchestra.run.smithi090.stdout:47244640300 2024-04-24T14:35:40.265 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640301 got 47244640300 for osd.4 2024-04-24T14:35:40.276 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640301 got 47244640300 for osd.2 2024-04-24T14:35:41.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T14:35:41.135 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T14:35:41.185 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T14:35:41.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T14:35:41.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T14:35:41.277 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T14:35:41.486 INFO:teuthology.orchestra.run.smithi090.stdout:47244640300 2024-04-24T14:35:41.503 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640301 got 47244640300 for osd.5 2024-04-24T14:35:41.506 INFO:teuthology.orchestra.run.smithi090.stdout:47244640298 2024-04-24T14:35:41.519 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640299 got 47244640298 for osd.3 2024-04-24T14:35:41.550 INFO:teuthology.orchestra.run.smithi090.stdout:47244640299 2024-04-24T14:35:41.569 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640300 got 47244640299 for osd.0 2024-04-24T14:35:41.606 INFO:teuthology.orchestra.run.smithi090.stdout:47244640300 2024-04-24T14:35:41.619 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640301 got 47244640300 for osd.4 2024-04-24T14:35:41.625 INFO:teuthology.orchestra.run.smithi090.stdout:283467841556 2024-04-24T14:35:41.640 INFO:tasks.ceph.ceph_manager.ceph:need seq 283467841557 got 283467841556 for osd.1 2024-04-24T14:35:41.678 INFO:teuthology.orchestra.run.smithi090.stdout:47244640300 2024-04-24T14:35:41.692 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640301 got 47244640300 for osd.2 2024-04-24T14:35:42.505 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T14:35:42.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T14:35:42.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T14:35:42.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T14:35:42.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T14:35:42.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T14:35:42.873 INFO:teuthology.orchestra.run.smithi090.stdout:47244640300 2024-04-24T14:35:42.889 INFO:teuthology.orchestra.run.smithi090.stdout:47244640301 2024-04-24T14:35:42.890 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640299 got 47244640300 for osd.3 2024-04-24T14:35:42.890 DEBUG:teuthology.parallel:result is None 2024-04-24T14:35:42.906 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640301 got 47244640301 for osd.5 2024-04-24T14:35:42.906 DEBUG:teuthology.parallel:result is None 2024-04-24T14:35:42.939 INFO:teuthology.orchestra.run.smithi090.stdout:47244640302 2024-04-24T14:35:42.951 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640301 got 47244640302 for osd.4 2024-04-24T14:35:42.952 DEBUG:teuthology.parallel:result is None 2024-04-24T14:35:42.959 INFO:teuthology.orchestra.run.smithi090.stdout:47244640301 2024-04-24T14:35:42.972 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640300 got 47244640301 for osd.0 2024-04-24T14:35:42.972 DEBUG:teuthology.parallel:result is None 2024-04-24T14:35:43.011 INFO:teuthology.orchestra.run.smithi090.stdout:283467841558 2024-04-24T14:35:43.024 INFO:tasks.ceph.ceph_manager.ceph:need seq 283467841557 got 283467841558 for osd.1 2024-04-24T14:35:43.025 DEBUG:teuthology.parallel:result is None 2024-04-24T14:35:43.039 INFO:teuthology.orchestra.run.smithi090.stdout:47244640302 2024-04-24T14:35:43.059 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640301 got 47244640302 for osd.2 2024-04-24T14:35:43.059 DEBUG:teuthology.parallel:result is None 2024-04-24T14:35:43.059 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-24T14:35:43.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:35:43.395 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:43.397 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:35:43.409 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-04-24T14:35:43.225988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80282741,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":46,"num_read":8344,"num_read_kb":182682,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":487769303,"num_keys_recovered":143,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1267,"ondisk_log_size":1267,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1975944,"kb_used_data":166564,"kb_used_omap":12,"kb_used_meta":1809267,"kb_avail":564255096,"statfs":{"total":579820584960,"available":577797218304,"internally_reserved":0,"allocated":170561536,"data_stored":164764697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13063,"internal_metadata":1852689657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-28,"num_bytes_recovered":16777275,"num_keys_recovered":2,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001716"},"pg_stats":[{"pgid":"1.0","version":"14'117","reported_seq":281,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981208+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:35:38.981208+0000","last_peered":"2024-04-24T14:35:38.981208+0000","last_clean":"2024-04-24T14:35:38.981208+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:35:38.981208+0000","last_undegraded":"2024-04-24T14:35:38.981208+0000","last_fullsized":"2024-04-24T14:35:38.981208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"49'3","reported_seq":68,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.984346+0000","last_change":"2024-04-24T14:35:19.912506+0000","last_active":"2024-04-24T14:35:38.984346+0000","last_peered":"2024-04-24T14:35:38.984346+0000","last_clean":"2024-04-24T14:35:38.984346+0000","last_became_active":"2024-04-24T14:35:04.316770+0000","last_became_peered":"2024-04-24T14:35:04.316770+0000","last_unstale":"2024-04-24T14:35:38.984346+0000","last_undegraded":"2024-04-24T14:35:38.984346+0000","last_fullsized":"2024-04-24T14:35:38.984346+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:06.338365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058253000000000005,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"119'187","reported_seq":609,"reported_epoch":137,"state":"peering","last_fresh":"2024-04-24T14:35:38.981170+0000","last_change":"2024-04-24T14:35:38.981539+0000","last_active":"2024-04-24T14:35:26.865729+0000","last_peered":"2024-04-24T14:35:26.670652+0000","last_clean":"2024-04-24T14:35:26.670652+0000","last_became_active":"2024-04-24T14:35:04.319930+0000","last_became_peered":"2024-04-24T14:35:04.319930+0000","last_unstale":"2024-04-24T14:35:38.981170+0000","last_undegraded":"2024-04-24T14:35:38.981170+0000","last_fullsized":"2024-04-24T14:35:38.981170+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:59:21.394861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013315029999999999,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":14,"num_read":1693,"num_read_kb":48628,"num_write":128,"num_write_kb":32867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":104988999,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"119'9","reported_seq":229,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981136+0000","last_change":"2024-04-24T14:35:28.058638+0000","last_active":"2024-04-24T14:35:38.981136+0000","last_peered":"2024-04-24T14:35:38.981136+0000","last_clean":"2024-04-24T14:35:38.981136+0000","last_became_active":"2024-04-24T14:35:27.881143+0000","last_became_peered":"2024-04-24T14:35:27.881143+0000","last_unstale":"2024-04-24T14:35:38.981136+0000","last_undegraded":"2024-04-24T14:35:38.981136+0000","last_fullsized":"2024-04-24T14:35:38.981136+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:08.850822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076176800000000002,"stat_sum":{"num_bytes":78,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":8,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165902,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"117'453","reported_seq":3073,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.982741+0000","last_change":"2024-04-24T14:35:29.912112+0000","last_active":"2024-04-24T14:35:38.982741+0000","last_peered":"2024-04-24T14:35:38.982741+0000","last_clean":"2024-04-24T14:35:38.982741+0000","last_became_active":"2024-04-24T14:35:29.910973+0000","last_became_peered":"2024-04-24T14:35:29.910973+0000","last_unstale":"2024-04-24T14:35:38.982741+0000","last_undegraded":"2024-04-24T14:35:38.982741+0000","last_fullsized":"2024-04-24T14:35:38.982741+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:47:08.597146+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0051668080000000002,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":11,"num_read":4739,"num_read_kb":77733,"num_write":290,"num_write_kb":56447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":25165862,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"119'7","reported_seq":235,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.988225+0000","last_change":"2024-04-24T14:35:27.935655+0000","last_active":"2024-04-24T14:35:38.988225+0000","last_peered":"2024-04-24T14:35:38.988225+0000","last_clean":"2024-04-24T14:35:38.988225+0000","last_became_active":"2024-04-24T14:35:27.884423+0000","last_became_peered":"2024-04-24T14:35:27.884423+0000","last_unstale":"2024-04-24T14:35:38.988225+0000","last_undegraded":"2024-04-24T14:35:38.988225+0000","last_fullsized":"2024-04-24T14:35:38.988225+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:30.623924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085674600000000003,"stat_sum":{"num_bytes":4194380,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":6,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":20971710,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"119'296","reported_seq":821,"reported_epoch":137,"state":"peering","last_fresh":"2024-04-24T14:35:38.981710+0000","last_change":"2024-04-24T14:35:38.982462+0000","last_active":"2024-04-24T14:35:26.874575+0000","last_peered":"2024-04-24T14:35:25.865537+0000","last_clean":"2024-04-24T14:35:25.865537+0000","last_became_active":"2024-04-24T14:34:44.841882+0000","last_became_peered":"2024-04-24T14:34:44.841882+0000","last_unstale":"2024-04-24T14:35:38.981710+0000","last_undegraded":"2024-04-24T14:35:38.981710+0000","last_fullsized":"2024-04-24T14:35:38.981710+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:38:50.903510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014897840000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1035,"num_read_kb":38768,"num_write":359,"num_write_kb":34059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":43040844,"num_keys_recovered":6,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"119'8","reported_seq":103,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.985758+0000","last_change":"2024-04-24T14:35:28.046471+0000","last_active":"2024-04-24T14:35:38.985758+0000","last_peered":"2024-04-24T14:35:38.985758+0000","last_clean":"2024-04-24T14:35:38.985758+0000","last_became_active":"2024-04-24T14:35:27.885728+0000","last_became_peered":"2024-04-24T14:35:27.885728+0000","last_unstale":"2024-04-24T14:35:38.985758+0000","last_undegraded":"2024-04-24T14:35:38.985758+0000","last_fullsized":"2024-04-24T14:35:38.985758+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:13.379266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086492199999999996,"stat_sum":{"num_bytes":16777294,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":14,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":100663374,"num_keys_recovered":21,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"119'147","reported_seq":271,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:39.648487+0000","last_change":"2024-04-24T14:35:19.849974+0000","last_active":"2024-04-24T14:35:39.648487+0000","last_peered":"2024-04-24T14:35:39.648487+0000","last_clean":"2024-04-24T14:35:39.648487+0000","last_became_active":"2024-04-24T14:34:06.852598+0000","last_became_peered":"2024-04-24T14:34:06.852598+0000","last_unstale":"2024-04-24T14:35:39.648487+0000","last_undegraded":"2024-04-24T14:35:39.648487+0000","last_fullsized":"2024-04-24T14:35:39.648487+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:01:25.107060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00154912,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":10,"num_read":707,"num_read_kb":17181,"num_write":118,"num_write_kb":16433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777362,"num_keys_recovered":17,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"119'17","reported_seq":40,"reported_epoch":137,"state":"peering","last_fresh":"2024-04-24T14:35:38.982990+0000","last_change":"2024-04-24T14:35:38.983890+0000","last_active":"2024-04-24T14:35:38.982928+0000","last_peered":"2024-04-24T14:35:28.015751+0000","last_clean":"2024-04-24T14:35:28.015751+0000","last_became_active":"2024-04-24T14:35:27.882559+0000","last_became_peered":"2024-04-24T14:35:27.882559+0000","last_unstale":"2024-04-24T14:35:38.982990+0000","last_undegraded":"2024-04-24T14:35:38.982990+0000","last_fullsized":"2024-04-24T14:35:38.982990+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:18:38.205103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061472399999999998,"stat_sum":{"num_bytes":20971636,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":16,"num_read_kb":135,"num_write":17,"num_write_kb":36868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":58720410,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"119'8","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981160+0000","last_change":"2024-04-24T14:35:19.843526+0000","last_active":"2024-04-24T14:35:38.981160+0000","last_peered":"2024-04-24T14:35:38.981160+0000","last_clean":"2024-04-24T14:35:38.981160+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:35:38.981160+0000","last_undegraded":"2024-04-24T14:35:38.981160+0000","last_fullsized":"2024-04-24T14:35:38.981160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001227351,"stat_sum":{"num_bytes":20971598,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":128,"num_write":6,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"119'7","reported_seq":141,"reported_epoch":137,"state":"peering","last_fresh":"2024-04-24T14:35:38.983609+0000","last_change":"2024-04-24T14:35:38.985054+0000","last_active":"2024-04-24T14:35:26.870997+0000","last_peered":"2024-04-24T14:35:25.863268+0000","last_clean":"2024-04-24T14:35:25.863268+0000","last_became_active":"2024-04-24T14:34:42.208652+0000","last_became_peered":"2024-04-24T14:34:42.208652+0000","last_unstale":"2024-04-24T14:35:38.983609+0000","last_undegraded":"2024-04-24T14:35:38.983609+0000","last_fullsized":"2024-04-24T14:35:38.983609+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:25:31.294378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013719380000000001,"stat_sum":{"num_bytes":4194342,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165862,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"119'8","reported_seq":211,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981177+0000","last_change":"2024-04-24T14:35:19.842955+0000","last_active":"2024-04-24T14:35:38.981177+0000","last_peered":"2024-04-24T14:35:38.981177+0000","last_clean":"2024-04-24T14:35:38.981177+0000","last_became_active":"2024-04-24T14:34:06.852864+0000","last_became_peered":"2024-04-24T14:34:06.852864+0000","last_unstale":"2024-04-24T14:35:38.981177+0000","last_undegraded":"2024-04-24T14:35:38.981177+0000","last_fullsized":"2024-04-24T14:35:38.981177+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:42.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000833036,"stat_sum":{"num_bytes":114,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":8,"num_write_kb":8196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":46,"num_read":8174,"num_read_kb":182310,"num_write":895,"num_write_kb":139806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":189973067,"num_keys_recovered":73,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1083,"ondisk_log_size":1083,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":79692354,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":54,"num_read_kb":272,"num_write":69,"num_write_kb":131096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":297796236,"num_keys_recovered":70,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159473664,"data_stored":159384708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640302,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331744,"kb_used_data":30292,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040096,"statfs":{"total":96636764160,"available":96297058304,"internally_reserved":0,"allocated":31019008,"data_stored":30060920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79600000000000004,"15min":0.81299999999999994},"min":{"1min":0.61299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0529999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.54800000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.23,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.024}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.82399999999999995}]},{"osd":2,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.83799999999999997},"min":{"1min":0.54100000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.026,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83899999999999997,"15min":0.85599999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0860000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.85599999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79200000000000004,"15min":0.82099999999999995},"min":{"1min":0.52000000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.133,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.622},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.82299999999999995,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.99299999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.89000000000000001}]},{"osd":4,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.86099999999999999,"15min":0.873},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1259999999999999,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.746},{"interface":"front","average":{"1min":0.873,"5min":0.93799999999999994,"15min":0.97899999999999998},"min":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.103,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.71699999999999997}]}]},{"osd":4,"up_from":11,"seq":47244640302,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1662976,"data_stored":700809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.78000000000000003},"min":{"1min":0.49399999999999999,"5min":0.379,"15min":0.379},"max":{"1min":1.0800000000000001,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.60499999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0049999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.503}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.61699999999999999}]},{"osd":2,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.83499999999999996,"15min":0.86099999999999999},"min":{"1min":0.63700000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.0189999999999999,"5min":1.113,"15min":1.113},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.64000000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0589999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83099999999999996,"15min":0.85099999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1970000000000001,"5min":1.931,"15min":1.931},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.85699999999999998,"15min":0.873},"min":{"1min":0.61499999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.161,"5min":1.591,"15min":1.591},"last":0.82799999999999996}]},{"osd":5,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.79300000000000004,"15min":0.78700000000000003},"min":{"1min":0.57999999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0629999999999999,"5min":1.087,"15min":1.087},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.90800000000000003,"15min":0.92700000000000005},"min":{"1min":0.57999999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.78700000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640300,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328252,"kb_used_data":26792,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94043588,"statfs":{"total":96636764160,"available":96300634112,"internally_reserved":0,"allocated":27435008,"data_stored":26457195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.753,"15min":0.77300000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0009999999999999,"5min":1.282,"15min":1.282},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80300000000000005,"15min":0.80100000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.99199999999999999}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":1.105},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.70099999999999996}]},{"osd":2,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84999999999999998,"15min":0.872},"min":{"1min":0.64900000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.006,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.91900000000000004,"15min":0.93600000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.238,"5min":1.337,"15min":1.337},"last":1.1759999999999999}]},{"osd":4,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.63200000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.52000000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.84399999999999997}]},{"osd":5,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79400000000000004,"15min":0.78800000000000003},"min":{"1min":0.625,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1200000000000001,"5min":1.367,"15min":1.367},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.871,"15min":0.872},"min":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.4199999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.78600000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640302,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319456,"kb_used_data":18004,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94052384,"statfs":{"total":96636764160,"available":96309641216,"internally_reserved":0,"allocated":18436096,"data_stored":17477987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85899999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60599999999999998},"min":{"1min":0.36299999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.871,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.45700000000000002}]},{"osd":1,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.35299999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.83899999999999997,"15min":0.85299999999999998},"min":{"1min":0.64500000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.117,"5min":1.147,"15min":1.147},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.873,"15min":0.90500000000000003},"min":{"1min":0.64200000000000002,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.98799999999999999,"5min":1.77,"15min":1.77},"last":0.76700000000000002}]},{"osd":4,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.85599999999999998,"15min":0.85699999999999998},"min":{"1min":0.70799999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.88400000000000001,"15min":0.89600000000000002},"min":{"1min":0.76500000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":0.68600000000000005}]},{"osd":5,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91900000000000004,"15min":0.95499999999999996},"min":{"1min":0.623,"5min":0.62,"15min":0.62},"max":{"1min":1.123,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.91800000000000004,"15min":0.92600000000000005},"min":{"1min":0.77500000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.77900000000000003}]}]},{"osd":1,"up_from":66,"seq":283467841558,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344692,"kb_used_data":46688,"kb_used_omap":4,"kb_used_meta":297979,"kb_avail":94027148,"statfs":{"total":96636764160,"available":96283799552,"internally_reserved":0,"allocated":47808512,"data_stored":46833489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305131073},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":0.80500000000000005}]},{"osd":2,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":2.9180000000000001}]},{"osd":3,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":3.0550000000000002}]},{"osd":4,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.94899999999999995}]},{"osd":5,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":1.0329999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640301,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348716,"kb_used_data":43164,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94023124,"statfs":{"total":96636764160,"available":96279678976,"internally_reserved":0,"allocated":44199936,"data_stored":43234297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 14:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.66300000000000003}]},{"osd":2,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.316,"15min":0.316},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68000000000000005,"15min":0.67900000000000005},"min":{"1min":0.45800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.83799999999999997}]},{"osd":3,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.022,"5min":1.472,"15min":1.472},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81100000000000005,"15min":0.80700000000000005},"min":{"1min":0.51600000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1299999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.77900000000000003}]},{"osd":4,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.89400000000000002,"15min":0.91700000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0309999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.91800000000000004,"15min":0.91900000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2210000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.91800000000000004}]},{"osd":5,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.70299999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.256,"5min":1.641,"15min":1.641},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.92200000000000004,"15min":0.92800000000000005},"min":{"1min":0.66900000000000004,"5min":0.628,"15min":0.628},"max":{"1min":1.1359999999999999,"5min":1.534,"15min":1.534},"last":0.89000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46137730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16785408,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25190400,"data_stored":25166132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:35:43.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:35:43.688 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:43.691 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:35:43.702 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-04-24T14:35:43.225988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80282741,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":46,"num_read":8344,"num_read_kb":182682,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":487769303,"num_keys_recovered":143,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1267,"ondisk_log_size":1267,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1975944,"kb_used_data":166564,"kb_used_omap":12,"kb_used_meta":1809267,"kb_avail":564255096,"statfs":{"total":579820584960,"available":577797218304,"internally_reserved":0,"allocated":170561536,"data_stored":164764697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13063,"internal_metadata":1852689657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-28,"num_bytes_recovered":16777275,"num_keys_recovered":2,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001716"},"pg_stats":[{"pgid":"1.0","version":"14'117","reported_seq":281,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981208+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:35:38.981208+0000","last_peered":"2024-04-24T14:35:38.981208+0000","last_clean":"2024-04-24T14:35:38.981208+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:35:38.981208+0000","last_undegraded":"2024-04-24T14:35:38.981208+0000","last_fullsized":"2024-04-24T14:35:38.981208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"49'3","reported_seq":68,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.984346+0000","last_change":"2024-04-24T14:35:19.912506+0000","last_active":"2024-04-24T14:35:38.984346+0000","last_peered":"2024-04-24T14:35:38.984346+0000","last_clean":"2024-04-24T14:35:38.984346+0000","last_became_active":"2024-04-24T14:35:04.316770+0000","last_became_peered":"2024-04-24T14:35:04.316770+0000","last_unstale":"2024-04-24T14:35:38.984346+0000","last_undegraded":"2024-04-24T14:35:38.984346+0000","last_fullsized":"2024-04-24T14:35:38.984346+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:06.338365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058253000000000005,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"119'187","reported_seq":609,"reported_epoch":137,"state":"peering","last_fresh":"2024-04-24T14:35:38.981170+0000","last_change":"2024-04-24T14:35:38.981539+0000","last_active":"2024-04-24T14:35:26.865729+0000","last_peered":"2024-04-24T14:35:26.670652+0000","last_clean":"2024-04-24T14:35:26.670652+0000","last_became_active":"2024-04-24T14:35:04.319930+0000","last_became_peered":"2024-04-24T14:35:04.319930+0000","last_unstale":"2024-04-24T14:35:38.981170+0000","last_undegraded":"2024-04-24T14:35:38.981170+0000","last_fullsized":"2024-04-24T14:35:38.981170+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:59:21.394861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013315029999999999,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":14,"num_read":1693,"num_read_kb":48628,"num_write":128,"num_write_kb":32867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":104988999,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"119'9","reported_seq":229,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981136+0000","last_change":"2024-04-24T14:35:28.058638+0000","last_active":"2024-04-24T14:35:38.981136+0000","last_peered":"2024-04-24T14:35:38.981136+0000","last_clean":"2024-04-24T14:35:38.981136+0000","last_became_active":"2024-04-24T14:35:27.881143+0000","last_became_peered":"2024-04-24T14:35:27.881143+0000","last_unstale":"2024-04-24T14:35:38.981136+0000","last_undegraded":"2024-04-24T14:35:38.981136+0000","last_fullsized":"2024-04-24T14:35:38.981136+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:08.850822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076176800000000002,"stat_sum":{"num_bytes":78,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":8,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165902,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"117'453","reported_seq":3073,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.982741+0000","last_change":"2024-04-24T14:35:29.912112+0000","last_active":"2024-04-24T14:35:38.982741+0000","last_peered":"2024-04-24T14:35:38.982741+0000","last_clean":"2024-04-24T14:35:38.982741+0000","last_became_active":"2024-04-24T14:35:29.910973+0000","last_became_peered":"2024-04-24T14:35:29.910973+0000","last_unstale":"2024-04-24T14:35:38.982741+0000","last_undegraded":"2024-04-24T14:35:38.982741+0000","last_fullsized":"2024-04-24T14:35:38.982741+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:47:08.597146+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0051668080000000002,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":11,"num_read":4739,"num_read_kb":77733,"num_write":290,"num_write_kb":56447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":25165862,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"119'7","reported_seq":235,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.988225+0000","last_change":"2024-04-24T14:35:27.935655+0000","last_active":"2024-04-24T14:35:38.988225+0000","last_peered":"2024-04-24T14:35:38.988225+0000","last_clean":"2024-04-24T14:35:38.988225+0000","last_became_active":"2024-04-24T14:35:27.884423+0000","last_became_peered":"2024-04-24T14:35:27.884423+0000","last_unstale":"2024-04-24T14:35:38.988225+0000","last_undegraded":"2024-04-24T14:35:38.988225+0000","last_fullsized":"2024-04-24T14:35:38.988225+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:30.623924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085674600000000003,"stat_sum":{"num_bytes":4194380,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":6,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":20971710,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"119'296","reported_seq":821,"reported_epoch":137,"state":"peering","last_fresh":"2024-04-24T14:35:38.981710+0000","last_change":"2024-04-24T14:35:38.982462+0000","last_active":"2024-04-24T14:35:26.874575+0000","last_peered":"2024-04-24T14:35:25.865537+0000","last_clean":"2024-04-24T14:35:25.865537+0000","last_became_active":"2024-04-24T14:34:44.841882+0000","last_became_peered":"2024-04-24T14:34:44.841882+0000","last_unstale":"2024-04-24T14:35:38.981710+0000","last_undegraded":"2024-04-24T14:35:38.981710+0000","last_fullsized":"2024-04-24T14:35:38.981710+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:38:50.903510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014897840000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1035,"num_read_kb":38768,"num_write":359,"num_write_kb":34059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":43040844,"num_keys_recovered":6,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"119'8","reported_seq":103,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.985758+0000","last_change":"2024-04-24T14:35:28.046471+0000","last_active":"2024-04-24T14:35:38.985758+0000","last_peered":"2024-04-24T14:35:38.985758+0000","last_clean":"2024-04-24T14:35:38.985758+0000","last_became_active":"2024-04-24T14:35:27.885728+0000","last_became_peered":"2024-04-24T14:35:27.885728+0000","last_unstale":"2024-04-24T14:35:38.985758+0000","last_undegraded":"2024-04-24T14:35:38.985758+0000","last_fullsized":"2024-04-24T14:35:38.985758+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:13.379266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086492199999999996,"stat_sum":{"num_bytes":16777294,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":14,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":100663374,"num_keys_recovered":21,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"119'147","reported_seq":271,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:39.648487+0000","last_change":"2024-04-24T14:35:19.849974+0000","last_active":"2024-04-24T14:35:39.648487+0000","last_peered":"2024-04-24T14:35:39.648487+0000","last_clean":"2024-04-24T14:35:39.648487+0000","last_became_active":"2024-04-24T14:34:06.852598+0000","last_became_peered":"2024-04-24T14:34:06.852598+0000","last_unstale":"2024-04-24T14:35:39.648487+0000","last_undegraded":"2024-04-24T14:35:39.648487+0000","last_fullsized":"2024-04-24T14:35:39.648487+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:01:25.107060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00154912,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":10,"num_read":707,"num_read_kb":17181,"num_write":118,"num_write_kb":16433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777362,"num_keys_recovered":17,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"119'17","reported_seq":40,"reported_epoch":137,"state":"peering","last_fresh":"2024-04-24T14:35:38.982990+0000","last_change":"2024-04-24T14:35:38.983890+0000","last_active":"2024-04-24T14:35:38.982928+0000","last_peered":"2024-04-24T14:35:28.015751+0000","last_clean":"2024-04-24T14:35:28.015751+0000","last_became_active":"2024-04-24T14:35:27.882559+0000","last_became_peered":"2024-04-24T14:35:27.882559+0000","last_unstale":"2024-04-24T14:35:38.982990+0000","last_undegraded":"2024-04-24T14:35:38.982990+0000","last_fullsized":"2024-04-24T14:35:38.982990+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:18:38.205103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061472399999999998,"stat_sum":{"num_bytes":20971636,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":16,"num_read_kb":135,"num_write":17,"num_write_kb":36868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":58720410,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"119'8","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981160+0000","last_change":"2024-04-24T14:35:19.843526+0000","last_active":"2024-04-24T14:35:38.981160+0000","last_peered":"2024-04-24T14:35:38.981160+0000","last_clean":"2024-04-24T14:35:38.981160+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:35:38.981160+0000","last_undegraded":"2024-04-24T14:35:38.981160+0000","last_fullsized":"2024-04-24T14:35:38.981160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001227351,"stat_sum":{"num_bytes":20971598,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":128,"num_write":6,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"119'7","reported_seq":141,"reported_epoch":137,"state":"peering","last_fresh":"2024-04-24T14:35:38.983609+0000","last_change":"2024-04-24T14:35:38.985054+0000","last_active":"2024-04-24T14:35:26.870997+0000","last_peered":"2024-04-24T14:35:25.863268+0000","last_clean":"2024-04-24T14:35:25.863268+0000","last_became_active":"2024-04-24T14:34:42.208652+0000","last_became_peered":"2024-04-24T14:34:42.208652+0000","last_unstale":"2024-04-24T14:35:38.983609+0000","last_undegraded":"2024-04-24T14:35:38.983609+0000","last_fullsized":"2024-04-24T14:35:38.983609+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:25:31.294378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013719380000000001,"stat_sum":{"num_bytes":4194342,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165862,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"119'8","reported_seq":211,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981177+0000","last_change":"2024-04-24T14:35:19.842955+0000","last_active":"2024-04-24T14:35:38.981177+0000","last_peered":"2024-04-24T14:35:38.981177+0000","last_clean":"2024-04-24T14:35:38.981177+0000","last_became_active":"2024-04-24T14:34:06.852864+0000","last_became_peered":"2024-04-24T14:34:06.852864+0000","last_unstale":"2024-04-24T14:35:38.981177+0000","last_undegraded":"2024-04-24T14:35:38.981177+0000","last_fullsized":"2024-04-24T14:35:38.981177+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:42.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000833036,"stat_sum":{"num_bytes":114,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":8,"num_write_kb":8196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":46,"num_read":8174,"num_read_kb":182310,"num_write":895,"num_write_kb":139806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":189973067,"num_keys_recovered":73,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1083,"ondisk_log_size":1083,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":79692354,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":54,"num_read_kb":272,"num_write":69,"num_write_kb":131096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":297796236,"num_keys_recovered":70,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159473664,"data_stored":159384708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640302,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331744,"kb_used_data":30292,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040096,"statfs":{"total":96636764160,"available":96297058304,"internally_reserved":0,"allocated":31019008,"data_stored":30060920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79600000000000004,"15min":0.81299999999999994},"min":{"1min":0.61299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0529999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.54800000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.23,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.024}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.82399999999999995}]},{"osd":2,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.83799999999999997},"min":{"1min":0.54100000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.026,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83899999999999997,"15min":0.85599999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0860000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.85599999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79200000000000004,"15min":0.82099999999999995},"min":{"1min":0.52000000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.133,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.622},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.82299999999999995,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.99299999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.89000000000000001}]},{"osd":4,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.86099999999999999,"15min":0.873},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1259999999999999,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.746},{"interface":"front","average":{"1min":0.873,"5min":0.93799999999999994,"15min":0.97899999999999998},"min":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.103,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.71699999999999997}]}]},{"osd":4,"up_from":11,"seq":47244640302,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1662976,"data_stored":700809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.78000000000000003},"min":{"1min":0.49399999999999999,"5min":0.379,"15min":0.379},"max":{"1min":1.0800000000000001,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.60499999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0049999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.503}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.61699999999999999}]},{"osd":2,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.83499999999999996,"15min":0.86099999999999999},"min":{"1min":0.63700000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.0189999999999999,"5min":1.113,"15min":1.113},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.64000000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0589999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83099999999999996,"15min":0.85099999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1970000000000001,"5min":1.931,"15min":1.931},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.85699999999999998,"15min":0.873},"min":{"1min":0.61499999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.161,"5min":1.591,"15min":1.591},"last":0.82799999999999996}]},{"osd":5,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.79300000000000004,"15min":0.78700000000000003},"min":{"1min":0.57999999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0629999999999999,"5min":1.087,"15min":1.087},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.90800000000000003,"15min":0.92700000000000005},"min":{"1min":0.57999999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.78700000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640300,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328252,"kb_used_data":26792,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94043588,"statfs":{"total":96636764160,"available":96300634112,"internally_reserved":0,"allocated":27435008,"data_stored":26457195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.753,"15min":0.77300000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0009999999999999,"5min":1.282,"15min":1.282},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80300000000000005,"15min":0.80100000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.99199999999999999}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":1.105},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.70099999999999996}]},{"osd":2,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84999999999999998,"15min":0.872},"min":{"1min":0.64900000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.006,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.91900000000000004,"15min":0.93600000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.238,"5min":1.337,"15min":1.337},"last":1.1759999999999999}]},{"osd":4,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.63200000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.52000000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.84399999999999997}]},{"osd":5,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79400000000000004,"15min":0.78800000000000003},"min":{"1min":0.625,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1200000000000001,"5min":1.367,"15min":1.367},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.871,"15min":0.872},"min":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.4199999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.78600000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640302,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319456,"kb_used_data":18004,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94052384,"statfs":{"total":96636764160,"available":96309641216,"internally_reserved":0,"allocated":18436096,"data_stored":17477987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85899999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60599999999999998},"min":{"1min":0.36299999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.871,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.45700000000000002}]},{"osd":1,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.35299999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.83899999999999997,"15min":0.85299999999999998},"min":{"1min":0.64500000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.117,"5min":1.147,"15min":1.147},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.873,"15min":0.90500000000000003},"min":{"1min":0.64200000000000002,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.98799999999999999,"5min":1.77,"15min":1.77},"last":0.76700000000000002}]},{"osd":4,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.85599999999999998,"15min":0.85699999999999998},"min":{"1min":0.70799999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.88400000000000001,"15min":0.89600000000000002},"min":{"1min":0.76500000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":0.68600000000000005}]},{"osd":5,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91900000000000004,"15min":0.95499999999999996},"min":{"1min":0.623,"5min":0.62,"15min":0.62},"max":{"1min":1.123,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.91800000000000004,"15min":0.92600000000000005},"min":{"1min":0.77500000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.77900000000000003}]}]},{"osd":1,"up_from":66,"seq":283467841558,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344692,"kb_used_data":46688,"kb_used_omap":4,"kb_used_meta":297979,"kb_avail":94027148,"statfs":{"total":96636764160,"available":96283799552,"internally_reserved":0,"allocated":47808512,"data_stored":46833489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305131073},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":0.80500000000000005}]},{"osd":2,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":2.9180000000000001}]},{"osd":3,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":3.0550000000000002}]},{"osd":4,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.94899999999999995}]},{"osd":5,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":1.0329999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640301,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348716,"kb_used_data":43164,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94023124,"statfs":{"total":96636764160,"available":96279678976,"internally_reserved":0,"allocated":44199936,"data_stored":43234297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 14:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.66300000000000003}]},{"osd":2,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.316,"15min":0.316},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68000000000000005,"15min":0.67900000000000005},"min":{"1min":0.45800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.83799999999999997}]},{"osd":3,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.022,"5min":1.472,"15min":1.472},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81100000000000005,"15min":0.80700000000000005},"min":{"1min":0.51600000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1299999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.77900000000000003}]},{"osd":4,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.89400000000000002,"15min":0.91700000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0309999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.91800000000000004,"15min":0.91900000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2210000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.91800000000000004}]},{"osd":5,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.70299999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.256,"5min":1.641,"15min":1.641},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.92200000000000004,"15min":0.92800000000000005},"min":{"1min":0.66900000000000004,"5min":0.628,"15min":0.628},"max":{"1min":1.1359999999999999,"5min":1.534,"15min":1.534},"last":0.89000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46137730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16785408,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25190400,"data_stored":25166132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:35:43.703 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-24T14:35:44.048 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:44.048 INFO:teuthology.orchestra.run.smithi090.stdout:{"fsid":"6706935e-d09a-472b-b839-321319fa9e54","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":215,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":138,"num_osds":6,"num_up_osds":6,"osd_up_since":1713969240,"num_in_osds":6,"osd_in_since":1713969326,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"peering","count":4}],"num_pgs":13,"num_pools":3,"num_objects":89,"data_bytes":80282741,"bytes_used":2023366656,"bytes_avail":577797218304,"bytes_total":579820584960,"inactive_pgs_ratio":0.30769231915473938,"recovering_objects_per_sec":0,"recovering_bytes_per_sec":2795412,"recovering_keys_per_sec":0,"num_objects_recovered":0,"num_bytes_recovered":16777275,"num_keys_recovered":2},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-24T14:35:01.215040+0000","services":{}},"progress_events":{}} 2024-04-24T14:35:44.067 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-04-24T14:35:44.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:35:44.350 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:44.352 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:35:44.363 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-04-24T14:35:43.225988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80282741,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":46,"num_read":8344,"num_read_kb":182682,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":487769303,"num_keys_recovered":143,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1267,"ondisk_log_size":1267,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1975944,"kb_used_data":166564,"kb_used_omap":12,"kb_used_meta":1809267,"kb_avail":564255096,"statfs":{"total":579820584960,"available":577797218304,"internally_reserved":0,"allocated":170561536,"data_stored":164764697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13063,"internal_metadata":1852689657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-28,"num_bytes_recovered":16777275,"num_keys_recovered":2,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001716"},"pg_stats":[{"pgid":"1.0","version":"14'117","reported_seq":281,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981208+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:35:38.981208+0000","last_peered":"2024-04-24T14:35:38.981208+0000","last_clean":"2024-04-24T14:35:38.981208+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:35:38.981208+0000","last_undegraded":"2024-04-24T14:35:38.981208+0000","last_fullsized":"2024-04-24T14:35:38.981208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"49'3","reported_seq":68,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.984346+0000","last_change":"2024-04-24T14:35:19.912506+0000","last_active":"2024-04-24T14:35:38.984346+0000","last_peered":"2024-04-24T14:35:38.984346+0000","last_clean":"2024-04-24T14:35:38.984346+0000","last_became_active":"2024-04-24T14:35:04.316770+0000","last_became_peered":"2024-04-24T14:35:04.316770+0000","last_unstale":"2024-04-24T14:35:38.984346+0000","last_undegraded":"2024-04-24T14:35:38.984346+0000","last_fullsized":"2024-04-24T14:35:38.984346+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:06.338365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058253000000000005,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"119'187","reported_seq":609,"reported_epoch":137,"state":"peering","last_fresh":"2024-04-24T14:35:38.981170+0000","last_change":"2024-04-24T14:35:38.981539+0000","last_active":"2024-04-24T14:35:26.865729+0000","last_peered":"2024-04-24T14:35:26.670652+0000","last_clean":"2024-04-24T14:35:26.670652+0000","last_became_active":"2024-04-24T14:35:04.319930+0000","last_became_peered":"2024-04-24T14:35:04.319930+0000","last_unstale":"2024-04-24T14:35:38.981170+0000","last_undegraded":"2024-04-24T14:35:38.981170+0000","last_fullsized":"2024-04-24T14:35:38.981170+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:59:21.394861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013315029999999999,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":14,"num_read":1693,"num_read_kb":48628,"num_write":128,"num_write_kb":32867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":104988999,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"119'9","reported_seq":229,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981136+0000","last_change":"2024-04-24T14:35:28.058638+0000","last_active":"2024-04-24T14:35:38.981136+0000","last_peered":"2024-04-24T14:35:38.981136+0000","last_clean":"2024-04-24T14:35:38.981136+0000","last_became_active":"2024-04-24T14:35:27.881143+0000","last_became_peered":"2024-04-24T14:35:27.881143+0000","last_unstale":"2024-04-24T14:35:38.981136+0000","last_undegraded":"2024-04-24T14:35:38.981136+0000","last_fullsized":"2024-04-24T14:35:38.981136+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:08.850822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076176800000000002,"stat_sum":{"num_bytes":78,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":8,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165902,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"117'453","reported_seq":3073,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.982741+0000","last_change":"2024-04-24T14:35:29.912112+0000","last_active":"2024-04-24T14:35:38.982741+0000","last_peered":"2024-04-24T14:35:38.982741+0000","last_clean":"2024-04-24T14:35:38.982741+0000","last_became_active":"2024-04-24T14:35:29.910973+0000","last_became_peered":"2024-04-24T14:35:29.910973+0000","last_unstale":"2024-04-24T14:35:38.982741+0000","last_undegraded":"2024-04-24T14:35:38.982741+0000","last_fullsized":"2024-04-24T14:35:38.982741+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:47:08.597146+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0051668080000000002,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":11,"num_read":4739,"num_read_kb":77733,"num_write":290,"num_write_kb":56447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":25165862,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"119'7","reported_seq":235,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.988225+0000","last_change":"2024-04-24T14:35:27.935655+0000","last_active":"2024-04-24T14:35:38.988225+0000","last_peered":"2024-04-24T14:35:38.988225+0000","last_clean":"2024-04-24T14:35:38.988225+0000","last_became_active":"2024-04-24T14:35:27.884423+0000","last_became_peered":"2024-04-24T14:35:27.884423+0000","last_unstale":"2024-04-24T14:35:38.988225+0000","last_undegraded":"2024-04-24T14:35:38.988225+0000","last_fullsized":"2024-04-24T14:35:38.988225+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:30.623924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085674600000000003,"stat_sum":{"num_bytes":4194380,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":6,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":20971710,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"119'296","reported_seq":821,"reported_epoch":137,"state":"peering","last_fresh":"2024-04-24T14:35:38.981710+0000","last_change":"2024-04-24T14:35:38.982462+0000","last_active":"2024-04-24T14:35:26.874575+0000","last_peered":"2024-04-24T14:35:25.865537+0000","last_clean":"2024-04-24T14:35:25.865537+0000","last_became_active":"2024-04-24T14:34:44.841882+0000","last_became_peered":"2024-04-24T14:34:44.841882+0000","last_unstale":"2024-04-24T14:35:38.981710+0000","last_undegraded":"2024-04-24T14:35:38.981710+0000","last_fullsized":"2024-04-24T14:35:38.981710+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:38:50.903510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014897840000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1035,"num_read_kb":38768,"num_write":359,"num_write_kb":34059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":43040844,"num_keys_recovered":6,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"119'8","reported_seq":103,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.985758+0000","last_change":"2024-04-24T14:35:28.046471+0000","last_active":"2024-04-24T14:35:38.985758+0000","last_peered":"2024-04-24T14:35:38.985758+0000","last_clean":"2024-04-24T14:35:38.985758+0000","last_became_active":"2024-04-24T14:35:27.885728+0000","last_became_peered":"2024-04-24T14:35:27.885728+0000","last_unstale":"2024-04-24T14:35:38.985758+0000","last_undegraded":"2024-04-24T14:35:38.985758+0000","last_fullsized":"2024-04-24T14:35:38.985758+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:13.379266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086492199999999996,"stat_sum":{"num_bytes":16777294,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":14,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":100663374,"num_keys_recovered":21,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"119'147","reported_seq":271,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:39.648487+0000","last_change":"2024-04-24T14:35:19.849974+0000","last_active":"2024-04-24T14:35:39.648487+0000","last_peered":"2024-04-24T14:35:39.648487+0000","last_clean":"2024-04-24T14:35:39.648487+0000","last_became_active":"2024-04-24T14:34:06.852598+0000","last_became_peered":"2024-04-24T14:34:06.852598+0000","last_unstale":"2024-04-24T14:35:39.648487+0000","last_undegraded":"2024-04-24T14:35:39.648487+0000","last_fullsized":"2024-04-24T14:35:39.648487+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:01:25.107060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00154912,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":10,"num_read":707,"num_read_kb":17181,"num_write":118,"num_write_kb":16433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777362,"num_keys_recovered":17,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"119'17","reported_seq":40,"reported_epoch":137,"state":"peering","last_fresh":"2024-04-24T14:35:38.982990+0000","last_change":"2024-04-24T14:35:38.983890+0000","last_active":"2024-04-24T14:35:38.982928+0000","last_peered":"2024-04-24T14:35:28.015751+0000","last_clean":"2024-04-24T14:35:28.015751+0000","last_became_active":"2024-04-24T14:35:27.882559+0000","last_became_peered":"2024-04-24T14:35:27.882559+0000","last_unstale":"2024-04-24T14:35:38.982990+0000","last_undegraded":"2024-04-24T14:35:38.982990+0000","last_fullsized":"2024-04-24T14:35:38.982990+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:18:38.205103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061472399999999998,"stat_sum":{"num_bytes":20971636,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":16,"num_read_kb":135,"num_write":17,"num_write_kb":36868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":58720410,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"119'8","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981160+0000","last_change":"2024-04-24T14:35:19.843526+0000","last_active":"2024-04-24T14:35:38.981160+0000","last_peered":"2024-04-24T14:35:38.981160+0000","last_clean":"2024-04-24T14:35:38.981160+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:35:38.981160+0000","last_undegraded":"2024-04-24T14:35:38.981160+0000","last_fullsized":"2024-04-24T14:35:38.981160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001227351,"stat_sum":{"num_bytes":20971598,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":128,"num_write":6,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"119'7","reported_seq":141,"reported_epoch":137,"state":"peering","last_fresh":"2024-04-24T14:35:38.983609+0000","last_change":"2024-04-24T14:35:38.985054+0000","last_active":"2024-04-24T14:35:26.870997+0000","last_peered":"2024-04-24T14:35:25.863268+0000","last_clean":"2024-04-24T14:35:25.863268+0000","last_became_active":"2024-04-24T14:34:42.208652+0000","last_became_peered":"2024-04-24T14:34:42.208652+0000","last_unstale":"2024-04-24T14:35:38.983609+0000","last_undegraded":"2024-04-24T14:35:38.983609+0000","last_fullsized":"2024-04-24T14:35:38.983609+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:25:31.294378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013719380000000001,"stat_sum":{"num_bytes":4194342,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165862,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"119'8","reported_seq":211,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981177+0000","last_change":"2024-04-24T14:35:19.842955+0000","last_active":"2024-04-24T14:35:38.981177+0000","last_peered":"2024-04-24T14:35:38.981177+0000","last_clean":"2024-04-24T14:35:38.981177+0000","last_became_active":"2024-04-24T14:34:06.852864+0000","last_became_peered":"2024-04-24T14:34:06.852864+0000","last_unstale":"2024-04-24T14:35:38.981177+0000","last_undegraded":"2024-04-24T14:35:38.981177+0000","last_fullsized":"2024-04-24T14:35:38.981177+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:42.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000833036,"stat_sum":{"num_bytes":114,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":8,"num_write_kb":8196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":46,"num_read":8174,"num_read_kb":182310,"num_write":895,"num_write_kb":139806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":189973067,"num_keys_recovered":73,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1083,"ondisk_log_size":1083,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":79692354,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":54,"num_read_kb":272,"num_write":69,"num_write_kb":131096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":297796236,"num_keys_recovered":70,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159473664,"data_stored":159384708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640302,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331744,"kb_used_data":30292,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040096,"statfs":{"total":96636764160,"available":96297058304,"internally_reserved":0,"allocated":31019008,"data_stored":30060920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79600000000000004,"15min":0.81299999999999994},"min":{"1min":0.61299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0529999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.54800000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.23,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.024}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.82399999999999995}]},{"osd":2,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.83799999999999997},"min":{"1min":0.54100000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.026,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83899999999999997,"15min":0.85599999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0860000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.85599999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79200000000000004,"15min":0.82099999999999995},"min":{"1min":0.52000000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.133,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.622},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.82299999999999995,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.99299999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.89000000000000001}]},{"osd":4,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.86099999999999999,"15min":0.873},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1259999999999999,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.746},{"interface":"front","average":{"1min":0.873,"5min":0.93799999999999994,"15min":0.97899999999999998},"min":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.103,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.71699999999999997}]}]},{"osd":4,"up_from":11,"seq":47244640302,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1662976,"data_stored":700809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.78000000000000003},"min":{"1min":0.49399999999999999,"5min":0.379,"15min":0.379},"max":{"1min":1.0800000000000001,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.60499999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0049999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.503}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.61699999999999999}]},{"osd":2,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.83499999999999996,"15min":0.86099999999999999},"min":{"1min":0.63700000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.0189999999999999,"5min":1.113,"15min":1.113},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.64000000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0589999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83099999999999996,"15min":0.85099999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1970000000000001,"5min":1.931,"15min":1.931},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.85699999999999998,"15min":0.873},"min":{"1min":0.61499999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.161,"5min":1.591,"15min":1.591},"last":0.82799999999999996}]},{"osd":5,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.79300000000000004,"15min":0.78700000000000003},"min":{"1min":0.57999999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0629999999999999,"5min":1.087,"15min":1.087},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.90800000000000003,"15min":0.92700000000000005},"min":{"1min":0.57999999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.78700000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640300,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328252,"kb_used_data":26792,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94043588,"statfs":{"total":96636764160,"available":96300634112,"internally_reserved":0,"allocated":27435008,"data_stored":26457195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.753,"15min":0.77300000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0009999999999999,"5min":1.282,"15min":1.282},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80300000000000005,"15min":0.80100000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.99199999999999999}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":1.105},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.70099999999999996}]},{"osd":2,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84999999999999998,"15min":0.872},"min":{"1min":0.64900000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.006,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.91900000000000004,"15min":0.93600000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.238,"5min":1.337,"15min":1.337},"last":1.1759999999999999}]},{"osd":4,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.63200000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.52000000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.84399999999999997}]},{"osd":5,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79400000000000004,"15min":0.78800000000000003},"min":{"1min":0.625,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1200000000000001,"5min":1.367,"15min":1.367},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.871,"15min":0.872},"min":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.4199999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.78600000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640302,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319456,"kb_used_data":18004,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94052384,"statfs":{"total":96636764160,"available":96309641216,"internally_reserved":0,"allocated":18436096,"data_stored":17477987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85899999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60599999999999998},"min":{"1min":0.36299999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.871,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.45700000000000002}]},{"osd":1,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.35299999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.83899999999999997,"15min":0.85299999999999998},"min":{"1min":0.64500000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.117,"5min":1.147,"15min":1.147},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.873,"15min":0.90500000000000003},"min":{"1min":0.64200000000000002,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.98799999999999999,"5min":1.77,"15min":1.77},"last":0.76700000000000002}]},{"osd":4,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.85599999999999998,"15min":0.85699999999999998},"min":{"1min":0.70799999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.88400000000000001,"15min":0.89600000000000002},"min":{"1min":0.76500000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":0.68600000000000005}]},{"osd":5,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91900000000000004,"15min":0.95499999999999996},"min":{"1min":0.623,"5min":0.62,"15min":0.62},"max":{"1min":1.123,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.91800000000000004,"15min":0.92600000000000005},"min":{"1min":0.77500000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.77900000000000003}]}]},{"osd":1,"up_from":66,"seq":283467841558,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344692,"kb_used_data":46688,"kb_used_omap":4,"kb_used_meta":297979,"kb_avail":94027148,"statfs":{"total":96636764160,"available":96283799552,"internally_reserved":0,"allocated":47808512,"data_stored":46833489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305131073},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":0.80500000000000005}]},{"osd":2,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":2.9180000000000001}]},{"osd":3,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":3.0550000000000002}]},{"osd":4,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.94899999999999995}]},{"osd":5,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":1.0329999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640301,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348716,"kb_used_data":43164,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94023124,"statfs":{"total":96636764160,"available":96279678976,"internally_reserved":0,"allocated":44199936,"data_stored":43234297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 14:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.66300000000000003}]},{"osd":2,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.316,"15min":0.316},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68000000000000005,"15min":0.67900000000000005},"min":{"1min":0.45800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.83799999999999997}]},{"osd":3,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.022,"5min":1.472,"15min":1.472},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81100000000000005,"15min":0.80700000000000005},"min":{"1min":0.51600000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1299999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.77900000000000003}]},{"osd":4,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.89400000000000002,"15min":0.91700000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0309999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.91800000000000004,"15min":0.91900000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2210000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.91800000000000004}]},{"osd":5,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.70299999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.256,"5min":1.641,"15min":1.641},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.92200000000000004,"15min":0.92800000000000005},"min":{"1min":0.66900000000000004,"5min":0.628,"15min":0.628},"max":{"1min":1.1359999999999999,"5min":1.534,"15min":1.534},"last":0.89000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46137730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16785408,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25190400,"data_stored":25166132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:35:47.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:35:47.664 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:47.666 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:35:47.678 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-04-24T14:35:47.227174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80282741,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":46,"num_read":8344,"num_read_kb":182682,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":487769303,"num_keys_recovered":143,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1267,"ondisk_log_size":1267,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1975948,"kb_used_data":166576,"kb_used_omap":12,"kb_used_meta":1809267,"kb_avail":564255092,"statfs":{"total":579820584960,"available":577797214208,"internally_reserved":0,"allocated":170573824,"data_stored":164769652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13063,"internal_metadata":1852689657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-28,"num_bytes_recovered":16777275,"num_keys_recovered":2,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.248732"},"pg_stats":[{"pgid":"1.0","version":"14'117","reported_seq":281,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981208+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:35:38.981208+0000","last_peered":"2024-04-24T14:35:38.981208+0000","last_clean":"2024-04-24T14:35:38.981208+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:35:38.981208+0000","last_undegraded":"2024-04-24T14:35:38.981208+0000","last_fullsized":"2024-04-24T14:35:38.981208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"49'3","reported_seq":68,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.984346+0000","last_change":"2024-04-24T14:35:19.912506+0000","last_active":"2024-04-24T14:35:38.984346+0000","last_peered":"2024-04-24T14:35:38.984346+0000","last_clean":"2024-04-24T14:35:38.984346+0000","last_became_active":"2024-04-24T14:35:04.316770+0000","last_became_peered":"2024-04-24T14:35:04.316770+0000","last_unstale":"2024-04-24T14:35:38.984346+0000","last_undegraded":"2024-04-24T14:35:38.984346+0000","last_fullsized":"2024-04-24T14:35:38.984346+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:06.338365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058253000000000005,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"119'187","reported_seq":617,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.003333+0000","last_change":"2024-04-24T14:35:40.003333+0000","last_active":"2024-04-24T14:35:40.003333+0000","last_peered":"2024-04-24T14:35:40.003333+0000","last_clean":"2024-04-24T14:35:40.003333+0000","last_became_active":"2024-04-24T14:35:40.002805+0000","last_became_peered":"2024-04-24T14:35:40.002805+0000","last_unstale":"2024-04-24T14:35:40.003333+0000","last_undegraded":"2024-04-24T14:35:40.003333+0000","last_fullsized":"2024-04-24T14:35:40.003333+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:02.689379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013315029999999999,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":14,"num_read":1693,"num_read_kb":48628,"num_write":128,"num_write_kb":32867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":104988999,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"119'9","reported_seq":229,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981136+0000","last_change":"2024-04-24T14:35:28.058638+0000","last_active":"2024-04-24T14:35:38.981136+0000","last_peered":"2024-04-24T14:35:38.981136+0000","last_clean":"2024-04-24T14:35:38.981136+0000","last_became_active":"2024-04-24T14:35:27.881143+0000","last_became_peered":"2024-04-24T14:35:27.881143+0000","last_unstale":"2024-04-24T14:35:38.981136+0000","last_undegraded":"2024-04-24T14:35:38.981136+0000","last_fullsized":"2024-04-24T14:35:38.981136+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:08.850822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076176800000000002,"stat_sum":{"num_bytes":78,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":8,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165902,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"117'453","reported_seq":3073,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.982741+0000","last_change":"2024-04-24T14:35:29.912112+0000","last_active":"2024-04-24T14:35:38.982741+0000","last_peered":"2024-04-24T14:35:38.982741+0000","last_clean":"2024-04-24T14:35:38.982741+0000","last_became_active":"2024-04-24T14:35:29.910973+0000","last_became_peered":"2024-04-24T14:35:29.910973+0000","last_unstale":"2024-04-24T14:35:38.982741+0000","last_undegraded":"2024-04-24T14:35:38.982741+0000","last_fullsized":"2024-04-24T14:35:38.982741+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:47:08.597146+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0051668080000000002,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":11,"num_read":4739,"num_read_kb":77733,"num_write":290,"num_write_kb":56447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":25165862,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"119'7","reported_seq":235,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.988225+0000","last_change":"2024-04-24T14:35:27.935655+0000","last_active":"2024-04-24T14:35:38.988225+0000","last_peered":"2024-04-24T14:35:38.988225+0000","last_clean":"2024-04-24T14:35:38.988225+0000","last_became_active":"2024-04-24T14:35:27.884423+0000","last_became_peered":"2024-04-24T14:35:27.884423+0000","last_unstale":"2024-04-24T14:35:38.988225+0000","last_undegraded":"2024-04-24T14:35:38.988225+0000","last_fullsized":"2024-04-24T14:35:38.988225+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:30.623924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085674600000000003,"stat_sum":{"num_bytes":4194380,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":6,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":20971710,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"119'296","reported_seq":829,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.007103+0000","last_change":"2024-04-24T14:35:40.007103+0000","last_active":"2024-04-24T14:35:40.007103+0000","last_peered":"2024-04-24T14:35:40.007103+0000","last_clean":"2024-04-24T14:35:40.007103+0000","last_became_active":"2024-04-24T14:35:40.006335+0000","last_became_peered":"2024-04-24T14:35:40.006335+0000","last_unstale":"2024-04-24T14:35:40.007103+0000","last_undegraded":"2024-04-24T14:35:40.007103+0000","last_fullsized":"2024-04-24T14:35:40.007103+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:49:50.086262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014897840000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1035,"num_read_kb":38768,"num_write":359,"num_write_kb":34059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":43040844,"num_keys_recovered":6,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"119'8","reported_seq":103,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.985758+0000","last_change":"2024-04-24T14:35:28.046471+0000","last_active":"2024-04-24T14:35:38.985758+0000","last_peered":"2024-04-24T14:35:38.985758+0000","last_clean":"2024-04-24T14:35:38.985758+0000","last_became_active":"2024-04-24T14:35:27.885728+0000","last_became_peered":"2024-04-24T14:35:27.885728+0000","last_unstale":"2024-04-24T14:35:38.985758+0000","last_undegraded":"2024-04-24T14:35:38.985758+0000","last_fullsized":"2024-04-24T14:35:38.985758+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:13.379266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086492199999999996,"stat_sum":{"num_bytes":16777294,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":14,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":100663374,"num_keys_recovered":21,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"119'147","reported_seq":271,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:39.648487+0000","last_change":"2024-04-24T14:35:19.849974+0000","last_active":"2024-04-24T14:35:39.648487+0000","last_peered":"2024-04-24T14:35:39.648487+0000","last_clean":"2024-04-24T14:35:39.648487+0000","last_became_active":"2024-04-24T14:34:06.852598+0000","last_became_peered":"2024-04-24T14:34:06.852598+0000","last_unstale":"2024-04-24T14:35:39.648487+0000","last_undegraded":"2024-04-24T14:35:39.648487+0000","last_fullsized":"2024-04-24T14:35:39.648487+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:01:25.107060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00154912,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":10,"num_read":707,"num_read_kb":17181,"num_write":118,"num_write_kb":16433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777362,"num_keys_recovered":17,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"119'17","reported_seq":48,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.002380+0000","last_change":"2024-04-24T14:35:40.002380+0000","last_active":"2024-04-24T14:35:40.002380+0000","last_peered":"2024-04-24T14:35:40.002380+0000","last_clean":"2024-04-24T14:35:40.002380+0000","last_became_active":"2024-04-24T14:35:40.001994+0000","last_became_peered":"2024-04-24T14:35:40.001994+0000","last_unstale":"2024-04-24T14:35:40.002380+0000","last_undegraded":"2024-04-24T14:35:40.002380+0000","last_fullsized":"2024-04-24T14:35:40.002380+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:09.304847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061472399999999998,"stat_sum":{"num_bytes":20971636,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":16,"num_read_kb":135,"num_write":17,"num_write_kb":36868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":58720410,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"119'8","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981160+0000","last_change":"2024-04-24T14:35:19.843526+0000","last_active":"2024-04-24T14:35:38.981160+0000","last_peered":"2024-04-24T14:35:38.981160+0000","last_clean":"2024-04-24T14:35:38.981160+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:35:38.981160+0000","last_undegraded":"2024-04-24T14:35:38.981160+0000","last_fullsized":"2024-04-24T14:35:38.981160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001227351,"stat_sum":{"num_bytes":20971598,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":128,"num_write":6,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"119'7","reported_seq":149,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.000611+0000","last_change":"2024-04-24T14:35:40.000611+0000","last_active":"2024-04-24T14:35:40.000611+0000","last_peered":"2024-04-24T14:35:40.000611+0000","last_clean":"2024-04-24T14:35:40.000611+0000","last_became_active":"2024-04-24T14:35:40.000089+0000","last_became_peered":"2024-04-24T14:35:40.000089+0000","last_unstale":"2024-04-24T14:35:40.000611+0000","last_undegraded":"2024-04-24T14:35:40.000611+0000","last_fullsized":"2024-04-24T14:35:40.000611+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:48:12.340747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013719380000000001,"stat_sum":{"num_bytes":4194342,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165862,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"119'8","reported_seq":211,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981177+0000","last_change":"2024-04-24T14:35:19.842955+0000","last_active":"2024-04-24T14:35:38.981177+0000","last_peered":"2024-04-24T14:35:38.981177+0000","last_clean":"2024-04-24T14:35:38.981177+0000","last_became_active":"2024-04-24T14:34:06.852864+0000","last_became_peered":"2024-04-24T14:34:06.852864+0000","last_unstale":"2024-04-24T14:35:38.981177+0000","last_undegraded":"2024-04-24T14:35:38.981177+0000","last_fullsized":"2024-04-24T14:35:38.981177+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:42.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000833036,"stat_sum":{"num_bytes":114,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":8,"num_write_kb":8196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":46,"num_read":8174,"num_read_kb":182310,"num_write":895,"num_write_kb":139806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":189973067,"num_keys_recovered":73,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1083,"ondisk_log_size":1083,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":79692354,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":54,"num_read_kb":272,"num_write":69,"num_write_kb":131096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":297796236,"num_keys_recovered":70,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159473664,"data_stored":159384708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331744,"kb_used_data":30292,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040096,"statfs":{"total":96636764160,"available":96297058304,"internally_reserved":0,"allocated":31019008,"data_stored":30060920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79600000000000004,"15min":0.81299999999999994},"min":{"1min":0.61299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0529999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.54800000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.23,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.98699999999999999}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.82899999999999996}]},{"osd":2,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.83799999999999997},"min":{"1min":0.54100000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.026,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.87},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83899999999999997,"15min":0.85599999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0860000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.80200000000000005}]},{"osd":3,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79200000000000004,"15min":0.82099999999999995},"min":{"1min":0.52000000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.133,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.82299999999999995,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.99299999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.65800000000000003}]},{"osd":4,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.86099999999999999,"15min":0.873},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1259999999999999,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.873,"5min":0.93799999999999994,"15min":0.97899999999999998},"min":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.103,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.95599999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1662976,"data_stored":700809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.78000000000000003},"min":{"1min":0.49399999999999999,"5min":0.379,"15min":0.379},"max":{"1min":1.0800000000000001,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.60499999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0049999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.84799999999999998}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.95999999999999996}]},{"osd":2,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.83499999999999996,"15min":0.86099999999999999},"min":{"1min":0.63700000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.0189999999999999,"5min":1.113,"15min":1.113},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.64000000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0589999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.99199999999999999}]},{"osd":3,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83099999999999996,"15min":0.85099999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1970000000000001,"5min":1.931,"15min":1.931},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.85699999999999998,"15min":0.873},"min":{"1min":0.61499999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.161,"5min":1.591,"15min":1.591},"last":0.59099999999999997}]},{"osd":5,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.79300000000000004,"15min":0.78700000000000003},"min":{"1min":0.57999999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0629999999999999,"5min":1.087,"15min":1.087},"last":0.753},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.90800000000000003,"15min":0.92700000000000005},"min":{"1min":0.57999999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.87}]}]},{"osd":3,"up_from":11,"seq":47244640301,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328252,"kb_used_data":26792,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94043588,"statfs":{"total":96636764160,"available":96300634112,"internally_reserved":0,"allocated":27435008,"data_stored":26457195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.753,"15min":0.77300000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0009999999999999,"5min":1.282,"15min":1.282},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80300000000000005,"15min":0.80100000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.93300000000000005}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.86599999999999999}]},{"osd":2,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84999999999999998,"15min":0.872},"min":{"1min":0.64900000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.006,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.91900000000000004,"15min":0.93600000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.238,"5min":1.337,"15min":1.337},"last":1.0389999999999999}]},{"osd":4,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.63200000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.52000000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.748}]},{"osd":5,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79400000000000004,"15min":0.78800000000000003},"min":{"1min":0.625,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1200000000000001,"5min":1.367,"15min":1.367},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.871,"15min":0.872},"min":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.4199999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.98899999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319456,"kb_used_data":18004,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94052384,"statfs":{"total":96636764160,"available":96309641216,"internally_reserved":0,"allocated":18436096,"data_stored":17477987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85899999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60599999999999998},"min":{"1min":0.36299999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.871,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.67100000000000004}]},{"osd":1,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.71799999999999997}]},{"osd":3,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.83899999999999997,"15min":0.85299999999999998},"min":{"1min":0.64500000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.117,"5min":1.147,"15min":1.147},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.873,"15min":0.90500000000000003},"min":{"1min":0.64200000000000002,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.98799999999999999,"5min":1.77,"15min":1.77},"last":0.94899999999999995}]},{"osd":4,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.85599999999999998,"15min":0.85699999999999998},"min":{"1min":0.70799999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.88400000000000001,"15min":0.89600000000000002},"min":{"1min":0.76500000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":0.88500000000000001}]},{"osd":5,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91900000000000004,"15min":0.95499999999999996},"min":{"1min":0.623,"5min":0.62,"15min":0.62},"max":{"1min":1.123,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.91800000000000004,"15min":0.92600000000000005},"min":{"1min":0.77500000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.97799999999999998}]}]},{"osd":1,"up_from":66,"seq":283467841559,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344696,"kb_used_data":46700,"kb_used_omap":4,"kb_used_meta":297979,"kb_avail":94027144,"statfs":{"total":96636764160,"available":96283795456,"internally_reserved":0,"allocated":47820800,"data_stored":46838444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305131073},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":0.54500000000000004}]},{"osd":2,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":0.60299999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.79800000000000004}]},{"osd":4,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.86199999999999999}]},{"osd":5,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.99399999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640302,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348716,"kb_used_data":43164,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94023124,"statfs":{"total":96636764160,"available":96279678976,"internally_reserved":0,"allocated":44199936,"data_stored":43234297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 14:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.41999999999999998}]},{"osd":2,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.316,"15min":0.316},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68000000000000005,"15min":0.67900000000000005},"min":{"1min":0.45800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.70299999999999996}]},{"osd":3,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.022,"5min":1.472,"15min":1.472},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81100000000000005,"15min":0.80700000000000005},"min":{"1min":0.51600000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1299999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.65300000000000002}]},{"osd":4,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.89400000000000002,"15min":0.91700000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0309999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.91800000000000004,"15min":0.91900000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2210000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.77900000000000003}]},{"osd":5,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.70299999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.256,"5min":1.641,"15min":1.641},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.92200000000000004,"15min":0.92800000000000005},"min":{"1min":0.66900000000000004,"5min":0.628,"15min":0.628},"max":{"1min":1.1359999999999999,"5min":1.534,"15min":1.534},"last":0.752}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46137730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16785408,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25190400,"data_stored":25166132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:35:47.680 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-24T14:35:47.680 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-24T14:35:47.991 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-24T14:35:48.000 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-04-24T14:35:48.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T14:35:48.289 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:48.289 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":138,"fsid":"6706935e-d09a-472b-b839-321319fa9e54","created":"2024-04-24T14:32:09.059925+0000","modified":"2024-04-24T14:35:39.977401+0000","last_up_change":"2024-04-24T14:34:00.856741+0000","last_in_change":"2024-04-24T14:35:26.859498+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T14:32:21.233022+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T14:32:26.609964+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"21","last_force_op_resend_prenautilus":"21","last_force_op_resend_preluminous":"21","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":119,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-04-24T14:32:33.987564+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"21","last_force_op_resend_prenautilus":"21","last_force_op_resend_preluminous":"21","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":119,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":2,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a2506b4f-3486-4f51-972f-1d9e4b015346","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":20309},{"type":"v1","addr":"172.21.15.90:6817","nonce":20309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":20309},{"type":"v1","addr":"172.21.15.90:6819","nonce":20309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":20309},{"type":"v1","addr":"172.21.15.90:6823","nonce":20309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":20309},{"type":"v1","addr":"172.21.15.90:6821","nonce":20309}]},"public_addr":"172.21.15.90:6817/20309","cluster_addr":"172.21.15.90:6819/20309","heartbeat_back_addr":"172.21.15.90:6823/20309","heartbeat_front_addr":"172.21.15.90:6821/20309","state":["exists","up"]},{"osd":1,"uuid":"9e17324f-a615-494f-ba0d-7c8aa8dac95c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":31,"up_from":66,"up_thru":137,"down_at":32,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":34413},{"type":"v1","addr":"172.21.15.90:6801","nonce":34413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":34413},{"type":"v1","addr":"172.21.15.90:6803","nonce":34413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":34413},{"type":"v1","addr":"172.21.15.90:6807","nonce":34413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":34413},{"type":"v1","addr":"172.21.15.90:6805","nonce":34413}]},"public_addr":"172.21.15.90:6801/34413","cluster_addr":"172.21.15.90:6803/34413","heartbeat_back_addr":"172.21.15.90:6807/34413","heartbeat_front_addr":"172.21.15.90:6805/34413","state":["exists","up"]},{"osd":2,"uuid":"f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":20310},{"type":"v1","addr":"172.21.15.90:6809","nonce":20310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":20310},{"type":"v1","addr":"172.21.15.90:6811","nonce":20310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":20310},{"type":"v1","addr":"172.21.15.90:6815","nonce":20310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":20310},{"type":"v1","addr":"172.21.15.90:6813","nonce":20310}]},"public_addr":"172.21.15.90:6809/20310","cluster_addr":"172.21.15.90:6811/20310","heartbeat_back_addr":"172.21.15.90:6815/20310","heartbeat_front_addr":"172.21.15.90:6813/20310","state":["exists","up"]},{"osd":3,"uuid":"509c16da-22b2-48e8-aa51-c320c8c4fd8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":20316},{"type":"v1","addr":"172.21.15.103:6801","nonce":20316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":20316},{"type":"v1","addr":"172.21.15.103:6803","nonce":20316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":20316},{"type":"v1","addr":"172.21.15.103:6807","nonce":20316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":20316},{"type":"v1","addr":"172.21.15.103:6805","nonce":20316}]},"public_addr":"172.21.15.103:6801/20316","cluster_addr":"172.21.15.103:6803/20316","heartbeat_back_addr":"172.21.15.103:6807/20316","heartbeat_front_addr":"172.21.15.103:6805/20316","state":["exists","up"]},{"osd":4,"uuid":"1462066b-4e59-46d4-9b85-1f61ee1d56a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":20318},{"type":"v1","addr":"172.21.15.103:6809","nonce":20318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":20318},{"type":"v1","addr":"172.21.15.103:6811","nonce":20318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":20318},{"type":"v1","addr":"172.21.15.103:6815","nonce":20318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":20318},{"type":"v1","addr":"172.21.15.103:6813","nonce":20318}]},"public_addr":"172.21.15.103:6809/20318","cluster_addr":"172.21.15.103:6811/20318","heartbeat_back_addr":"172.21.15.103:6815/20318","heartbeat_front_addr":"172.21.15.103:6813/20318","state":["exists","up"]},{"osd":5,"uuid":"4dc954a8-5f3c-4826-b9bc-98cf30519382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":20317},{"type":"v1","addr":"172.21.15.103:6817","nonce":20317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":20317},{"type":"v1","addr":"172.21.15.103:6819","nonce":20317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":20317},{"type":"v1","addr":"172.21.15.103:6823","nonce":20317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":20317},{"type":"v1","addr":"172.21.15.103:6821","nonce":20317}]},"public_addr":"172.21.15.103:6817/20317","cluster_addr":"172.21.15.103:6819/20317","heartbeat_back_addr":"172.21.15.103:6823/20317","heartbeat_front_addr":"172.21.15.103:6821/20317","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.792899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.580414+0000","dead_epoch":32},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.713336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.579638+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.570510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.736558+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T14:35:48.303 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T14:35:48.303 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:35:48.590 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:48.593 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:35:48.604 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-04-24T14:35:47.227174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80282741,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":46,"num_read":8344,"num_read_kb":182682,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":487769303,"num_keys_recovered":143,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1267,"ondisk_log_size":1267,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1975948,"kb_used_data":166576,"kb_used_omap":12,"kb_used_meta":1809267,"kb_avail":564255092,"statfs":{"total":579820584960,"available":577797214208,"internally_reserved":0,"allocated":170573824,"data_stored":164769652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13063,"internal_metadata":1852689657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-28,"num_bytes_recovered":16777275,"num_keys_recovered":2,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.248732"},"pg_stats":[{"pgid":"1.0","version":"14'117","reported_seq":281,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981208+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:35:38.981208+0000","last_peered":"2024-04-24T14:35:38.981208+0000","last_clean":"2024-04-24T14:35:38.981208+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:35:38.981208+0000","last_undegraded":"2024-04-24T14:35:38.981208+0000","last_fullsized":"2024-04-24T14:35:38.981208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"49'3","reported_seq":68,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.984346+0000","last_change":"2024-04-24T14:35:19.912506+0000","last_active":"2024-04-24T14:35:38.984346+0000","last_peered":"2024-04-24T14:35:38.984346+0000","last_clean":"2024-04-24T14:35:38.984346+0000","last_became_active":"2024-04-24T14:35:04.316770+0000","last_became_peered":"2024-04-24T14:35:04.316770+0000","last_unstale":"2024-04-24T14:35:38.984346+0000","last_undegraded":"2024-04-24T14:35:38.984346+0000","last_fullsized":"2024-04-24T14:35:38.984346+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:06.338365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058253000000000005,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"119'187","reported_seq":617,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.003333+0000","last_change":"2024-04-24T14:35:40.003333+0000","last_active":"2024-04-24T14:35:40.003333+0000","last_peered":"2024-04-24T14:35:40.003333+0000","last_clean":"2024-04-24T14:35:40.003333+0000","last_became_active":"2024-04-24T14:35:40.002805+0000","last_became_peered":"2024-04-24T14:35:40.002805+0000","last_unstale":"2024-04-24T14:35:40.003333+0000","last_undegraded":"2024-04-24T14:35:40.003333+0000","last_fullsized":"2024-04-24T14:35:40.003333+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:02.689379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013315029999999999,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":14,"num_read":1693,"num_read_kb":48628,"num_write":128,"num_write_kb":32867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":104988999,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"119'9","reported_seq":229,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981136+0000","last_change":"2024-04-24T14:35:28.058638+0000","last_active":"2024-04-24T14:35:38.981136+0000","last_peered":"2024-04-24T14:35:38.981136+0000","last_clean":"2024-04-24T14:35:38.981136+0000","last_became_active":"2024-04-24T14:35:27.881143+0000","last_became_peered":"2024-04-24T14:35:27.881143+0000","last_unstale":"2024-04-24T14:35:38.981136+0000","last_undegraded":"2024-04-24T14:35:38.981136+0000","last_fullsized":"2024-04-24T14:35:38.981136+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:08.850822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076176800000000002,"stat_sum":{"num_bytes":78,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":8,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165902,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"117'453","reported_seq":3073,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.982741+0000","last_change":"2024-04-24T14:35:29.912112+0000","last_active":"2024-04-24T14:35:38.982741+0000","last_peered":"2024-04-24T14:35:38.982741+0000","last_clean":"2024-04-24T14:35:38.982741+0000","last_became_active":"2024-04-24T14:35:29.910973+0000","last_became_peered":"2024-04-24T14:35:29.910973+0000","last_unstale":"2024-04-24T14:35:38.982741+0000","last_undegraded":"2024-04-24T14:35:38.982741+0000","last_fullsized":"2024-04-24T14:35:38.982741+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:47:08.597146+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0051668080000000002,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":11,"num_read":4739,"num_read_kb":77733,"num_write":290,"num_write_kb":56447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":25165862,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"119'7","reported_seq":235,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.988225+0000","last_change":"2024-04-24T14:35:27.935655+0000","last_active":"2024-04-24T14:35:38.988225+0000","last_peered":"2024-04-24T14:35:38.988225+0000","last_clean":"2024-04-24T14:35:38.988225+0000","last_became_active":"2024-04-24T14:35:27.884423+0000","last_became_peered":"2024-04-24T14:35:27.884423+0000","last_unstale":"2024-04-24T14:35:38.988225+0000","last_undegraded":"2024-04-24T14:35:38.988225+0000","last_fullsized":"2024-04-24T14:35:38.988225+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:30.623924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085674600000000003,"stat_sum":{"num_bytes":4194380,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":6,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":20971710,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"119'296","reported_seq":829,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.007103+0000","last_change":"2024-04-24T14:35:40.007103+0000","last_active":"2024-04-24T14:35:40.007103+0000","last_peered":"2024-04-24T14:35:40.007103+0000","last_clean":"2024-04-24T14:35:40.007103+0000","last_became_active":"2024-04-24T14:35:40.006335+0000","last_became_peered":"2024-04-24T14:35:40.006335+0000","last_unstale":"2024-04-24T14:35:40.007103+0000","last_undegraded":"2024-04-24T14:35:40.007103+0000","last_fullsized":"2024-04-24T14:35:40.007103+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:49:50.086262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014897840000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1035,"num_read_kb":38768,"num_write":359,"num_write_kb":34059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":43040844,"num_keys_recovered":6,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"119'8","reported_seq":103,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.985758+0000","last_change":"2024-04-24T14:35:28.046471+0000","last_active":"2024-04-24T14:35:38.985758+0000","last_peered":"2024-04-24T14:35:38.985758+0000","last_clean":"2024-04-24T14:35:38.985758+0000","last_became_active":"2024-04-24T14:35:27.885728+0000","last_became_peered":"2024-04-24T14:35:27.885728+0000","last_unstale":"2024-04-24T14:35:38.985758+0000","last_undegraded":"2024-04-24T14:35:38.985758+0000","last_fullsized":"2024-04-24T14:35:38.985758+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:13.379266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086492199999999996,"stat_sum":{"num_bytes":16777294,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":14,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":100663374,"num_keys_recovered":21,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"119'147","reported_seq":271,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:39.648487+0000","last_change":"2024-04-24T14:35:19.849974+0000","last_active":"2024-04-24T14:35:39.648487+0000","last_peered":"2024-04-24T14:35:39.648487+0000","last_clean":"2024-04-24T14:35:39.648487+0000","last_became_active":"2024-04-24T14:34:06.852598+0000","last_became_peered":"2024-04-24T14:34:06.852598+0000","last_unstale":"2024-04-24T14:35:39.648487+0000","last_undegraded":"2024-04-24T14:35:39.648487+0000","last_fullsized":"2024-04-24T14:35:39.648487+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:01:25.107060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00154912,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":10,"num_read":707,"num_read_kb":17181,"num_write":118,"num_write_kb":16433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777362,"num_keys_recovered":17,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"119'17","reported_seq":48,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.002380+0000","last_change":"2024-04-24T14:35:40.002380+0000","last_active":"2024-04-24T14:35:40.002380+0000","last_peered":"2024-04-24T14:35:40.002380+0000","last_clean":"2024-04-24T14:35:40.002380+0000","last_became_active":"2024-04-24T14:35:40.001994+0000","last_became_peered":"2024-04-24T14:35:40.001994+0000","last_unstale":"2024-04-24T14:35:40.002380+0000","last_undegraded":"2024-04-24T14:35:40.002380+0000","last_fullsized":"2024-04-24T14:35:40.002380+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:09.304847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061472399999999998,"stat_sum":{"num_bytes":20971636,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":16,"num_read_kb":135,"num_write":17,"num_write_kb":36868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":58720410,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"119'8","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981160+0000","last_change":"2024-04-24T14:35:19.843526+0000","last_active":"2024-04-24T14:35:38.981160+0000","last_peered":"2024-04-24T14:35:38.981160+0000","last_clean":"2024-04-24T14:35:38.981160+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:35:38.981160+0000","last_undegraded":"2024-04-24T14:35:38.981160+0000","last_fullsized":"2024-04-24T14:35:38.981160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001227351,"stat_sum":{"num_bytes":20971598,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":128,"num_write":6,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"119'7","reported_seq":149,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.000611+0000","last_change":"2024-04-24T14:35:40.000611+0000","last_active":"2024-04-24T14:35:40.000611+0000","last_peered":"2024-04-24T14:35:40.000611+0000","last_clean":"2024-04-24T14:35:40.000611+0000","last_became_active":"2024-04-24T14:35:40.000089+0000","last_became_peered":"2024-04-24T14:35:40.000089+0000","last_unstale":"2024-04-24T14:35:40.000611+0000","last_undegraded":"2024-04-24T14:35:40.000611+0000","last_fullsized":"2024-04-24T14:35:40.000611+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:48:12.340747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013719380000000001,"stat_sum":{"num_bytes":4194342,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165862,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"119'8","reported_seq":211,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981177+0000","last_change":"2024-04-24T14:35:19.842955+0000","last_active":"2024-04-24T14:35:38.981177+0000","last_peered":"2024-04-24T14:35:38.981177+0000","last_clean":"2024-04-24T14:35:38.981177+0000","last_became_active":"2024-04-24T14:34:06.852864+0000","last_became_peered":"2024-04-24T14:34:06.852864+0000","last_unstale":"2024-04-24T14:35:38.981177+0000","last_undegraded":"2024-04-24T14:35:38.981177+0000","last_fullsized":"2024-04-24T14:35:38.981177+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:42.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000833036,"stat_sum":{"num_bytes":114,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":8,"num_write_kb":8196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":46,"num_read":8174,"num_read_kb":182310,"num_write":895,"num_write_kb":139806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":189973067,"num_keys_recovered":73,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1083,"ondisk_log_size":1083,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":79692354,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":54,"num_read_kb":272,"num_write":69,"num_write_kb":131096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":297796236,"num_keys_recovered":70,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159473664,"data_stored":159384708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331744,"kb_used_data":30292,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040096,"statfs":{"total":96636764160,"available":96297058304,"internally_reserved":0,"allocated":31019008,"data_stored":30060920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79600000000000004,"15min":0.81299999999999994},"min":{"1min":0.61299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0529999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.54800000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.23,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.98699999999999999}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.82899999999999996}]},{"osd":2,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.83799999999999997},"min":{"1min":0.54100000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.026,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.87},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83899999999999997,"15min":0.85599999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0860000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.80200000000000005}]},{"osd":3,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79200000000000004,"15min":0.82099999999999995},"min":{"1min":0.52000000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.133,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.82299999999999995,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.99299999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.65800000000000003}]},{"osd":4,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.86099999999999999,"15min":0.873},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1259999999999999,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.873,"5min":0.93799999999999994,"15min":0.97899999999999998},"min":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.103,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.95599999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1662976,"data_stored":700809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.78000000000000003},"min":{"1min":0.49399999999999999,"5min":0.379,"15min":0.379},"max":{"1min":1.0800000000000001,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.60499999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0049999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.84799999999999998}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.95999999999999996}]},{"osd":2,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.83499999999999996,"15min":0.86099999999999999},"min":{"1min":0.63700000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.0189999999999999,"5min":1.113,"15min":1.113},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.64000000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0589999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.99199999999999999}]},{"osd":3,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83099999999999996,"15min":0.85099999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1970000000000001,"5min":1.931,"15min":1.931},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.85699999999999998,"15min":0.873},"min":{"1min":0.61499999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.161,"5min":1.591,"15min":1.591},"last":0.59099999999999997}]},{"osd":5,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.79300000000000004,"15min":0.78700000000000003},"min":{"1min":0.57999999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0629999999999999,"5min":1.087,"15min":1.087},"last":0.753},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.90800000000000003,"15min":0.92700000000000005},"min":{"1min":0.57999999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.87}]}]},{"osd":3,"up_from":11,"seq":47244640301,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328252,"kb_used_data":26792,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94043588,"statfs":{"total":96636764160,"available":96300634112,"internally_reserved":0,"allocated":27435008,"data_stored":26457195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.753,"15min":0.77300000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0009999999999999,"5min":1.282,"15min":1.282},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80300000000000005,"15min":0.80100000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.93300000000000005}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.86599999999999999}]},{"osd":2,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84999999999999998,"15min":0.872},"min":{"1min":0.64900000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.006,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.91900000000000004,"15min":0.93600000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.238,"5min":1.337,"15min":1.337},"last":1.0389999999999999}]},{"osd":4,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.63200000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.52000000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.748}]},{"osd":5,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79400000000000004,"15min":0.78800000000000003},"min":{"1min":0.625,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1200000000000001,"5min":1.367,"15min":1.367},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.871,"15min":0.872},"min":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.4199999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.98899999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319456,"kb_used_data":18004,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94052384,"statfs":{"total":96636764160,"available":96309641216,"internally_reserved":0,"allocated":18436096,"data_stored":17477987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85899999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60599999999999998},"min":{"1min":0.36299999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.871,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.67100000000000004}]},{"osd":1,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.71799999999999997}]},{"osd":3,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.83899999999999997,"15min":0.85299999999999998},"min":{"1min":0.64500000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.117,"5min":1.147,"15min":1.147},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.873,"15min":0.90500000000000003},"min":{"1min":0.64200000000000002,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.98799999999999999,"5min":1.77,"15min":1.77},"last":0.94899999999999995}]},{"osd":4,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.85599999999999998,"15min":0.85699999999999998},"min":{"1min":0.70799999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.88400000000000001,"15min":0.89600000000000002},"min":{"1min":0.76500000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":0.88500000000000001}]},{"osd":5,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91900000000000004,"15min":0.95499999999999996},"min":{"1min":0.623,"5min":0.62,"15min":0.62},"max":{"1min":1.123,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.91800000000000004,"15min":0.92600000000000005},"min":{"1min":0.77500000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.97799999999999998}]}]},{"osd":1,"up_from":66,"seq":283467841559,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344696,"kb_used_data":46700,"kb_used_omap":4,"kb_used_meta":297979,"kb_avail":94027144,"statfs":{"total":96636764160,"available":96283795456,"internally_reserved":0,"allocated":47820800,"data_stored":46838444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305131073},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":0.54500000000000004}]},{"osd":2,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":0.60299999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.79800000000000004}]},{"osd":4,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.86199999999999999}]},{"osd":5,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.99399999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640302,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348716,"kb_used_data":43164,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94023124,"statfs":{"total":96636764160,"available":96279678976,"internally_reserved":0,"allocated":44199936,"data_stored":43234297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 14:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.41999999999999998}]},{"osd":2,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.316,"15min":0.316},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68000000000000005,"15min":0.67900000000000005},"min":{"1min":0.45800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.70299999999999996}]},{"osd":3,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.022,"5min":1.472,"15min":1.472},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81100000000000005,"15min":0.80700000000000005},"min":{"1min":0.51600000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1299999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.65300000000000002}]},{"osd":4,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.89400000000000002,"15min":0.91700000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0309999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.91800000000000004,"15min":0.91900000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2210000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.77900000000000003}]},{"osd":5,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.70299999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.256,"5min":1.641,"15min":1.641},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.92200000000000004,"15min":0.92800000000000005},"min":{"1min":0.66900000000000004,"5min":0.628,"15min":0.628},"max":{"1min":1.1359999999999999,"5min":1.534,"15min":1.534},"last":0.752}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46137730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16785408,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25190400,"data_stored":25166132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:35:48.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:35:48.897 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:48.899 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:35:48.910 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-04-24T14:35:47.227174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80282741,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":46,"num_read":8344,"num_read_kb":182682,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":487769303,"num_keys_recovered":143,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1267,"ondisk_log_size":1267,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1975948,"kb_used_data":166576,"kb_used_omap":12,"kb_used_meta":1809267,"kb_avail":564255092,"statfs":{"total":579820584960,"available":577797214208,"internally_reserved":0,"allocated":170573824,"data_stored":164769652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13063,"internal_metadata":1852689657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-28,"num_bytes_recovered":16777275,"num_keys_recovered":2,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.248732"},"pg_stats":[{"pgid":"1.0","version":"14'117","reported_seq":281,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981208+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:35:38.981208+0000","last_peered":"2024-04-24T14:35:38.981208+0000","last_clean":"2024-04-24T14:35:38.981208+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:35:38.981208+0000","last_undegraded":"2024-04-24T14:35:38.981208+0000","last_fullsized":"2024-04-24T14:35:38.981208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"49'3","reported_seq":68,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.984346+0000","last_change":"2024-04-24T14:35:19.912506+0000","last_active":"2024-04-24T14:35:38.984346+0000","last_peered":"2024-04-24T14:35:38.984346+0000","last_clean":"2024-04-24T14:35:38.984346+0000","last_became_active":"2024-04-24T14:35:04.316770+0000","last_became_peered":"2024-04-24T14:35:04.316770+0000","last_unstale":"2024-04-24T14:35:38.984346+0000","last_undegraded":"2024-04-24T14:35:38.984346+0000","last_fullsized":"2024-04-24T14:35:38.984346+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:06.338365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058253000000000005,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"119'187","reported_seq":617,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.003333+0000","last_change":"2024-04-24T14:35:40.003333+0000","last_active":"2024-04-24T14:35:40.003333+0000","last_peered":"2024-04-24T14:35:40.003333+0000","last_clean":"2024-04-24T14:35:40.003333+0000","last_became_active":"2024-04-24T14:35:40.002805+0000","last_became_peered":"2024-04-24T14:35:40.002805+0000","last_unstale":"2024-04-24T14:35:40.003333+0000","last_undegraded":"2024-04-24T14:35:40.003333+0000","last_fullsized":"2024-04-24T14:35:40.003333+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:02.689379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013315029999999999,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":14,"num_read":1693,"num_read_kb":48628,"num_write":128,"num_write_kb":32867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":104988999,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"119'9","reported_seq":229,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981136+0000","last_change":"2024-04-24T14:35:28.058638+0000","last_active":"2024-04-24T14:35:38.981136+0000","last_peered":"2024-04-24T14:35:38.981136+0000","last_clean":"2024-04-24T14:35:38.981136+0000","last_became_active":"2024-04-24T14:35:27.881143+0000","last_became_peered":"2024-04-24T14:35:27.881143+0000","last_unstale":"2024-04-24T14:35:38.981136+0000","last_undegraded":"2024-04-24T14:35:38.981136+0000","last_fullsized":"2024-04-24T14:35:38.981136+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:08.850822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076176800000000002,"stat_sum":{"num_bytes":78,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":8,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165902,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"117'453","reported_seq":3073,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.982741+0000","last_change":"2024-04-24T14:35:29.912112+0000","last_active":"2024-04-24T14:35:38.982741+0000","last_peered":"2024-04-24T14:35:38.982741+0000","last_clean":"2024-04-24T14:35:38.982741+0000","last_became_active":"2024-04-24T14:35:29.910973+0000","last_became_peered":"2024-04-24T14:35:29.910973+0000","last_unstale":"2024-04-24T14:35:38.982741+0000","last_undegraded":"2024-04-24T14:35:38.982741+0000","last_fullsized":"2024-04-24T14:35:38.982741+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:47:08.597146+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0051668080000000002,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":11,"num_read":4739,"num_read_kb":77733,"num_write":290,"num_write_kb":56447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":25165862,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"119'7","reported_seq":235,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.988225+0000","last_change":"2024-04-24T14:35:27.935655+0000","last_active":"2024-04-24T14:35:38.988225+0000","last_peered":"2024-04-24T14:35:38.988225+0000","last_clean":"2024-04-24T14:35:38.988225+0000","last_became_active":"2024-04-24T14:35:27.884423+0000","last_became_peered":"2024-04-24T14:35:27.884423+0000","last_unstale":"2024-04-24T14:35:38.988225+0000","last_undegraded":"2024-04-24T14:35:38.988225+0000","last_fullsized":"2024-04-24T14:35:38.988225+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:30.623924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085674600000000003,"stat_sum":{"num_bytes":4194380,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":6,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":20971710,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"119'296","reported_seq":829,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.007103+0000","last_change":"2024-04-24T14:35:40.007103+0000","last_active":"2024-04-24T14:35:40.007103+0000","last_peered":"2024-04-24T14:35:40.007103+0000","last_clean":"2024-04-24T14:35:40.007103+0000","last_became_active":"2024-04-24T14:35:40.006335+0000","last_became_peered":"2024-04-24T14:35:40.006335+0000","last_unstale":"2024-04-24T14:35:40.007103+0000","last_undegraded":"2024-04-24T14:35:40.007103+0000","last_fullsized":"2024-04-24T14:35:40.007103+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:49:50.086262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014897840000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1035,"num_read_kb":38768,"num_write":359,"num_write_kb":34059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":43040844,"num_keys_recovered":6,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"119'8","reported_seq":103,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.985758+0000","last_change":"2024-04-24T14:35:28.046471+0000","last_active":"2024-04-24T14:35:38.985758+0000","last_peered":"2024-04-24T14:35:38.985758+0000","last_clean":"2024-04-24T14:35:38.985758+0000","last_became_active":"2024-04-24T14:35:27.885728+0000","last_became_peered":"2024-04-24T14:35:27.885728+0000","last_unstale":"2024-04-24T14:35:38.985758+0000","last_undegraded":"2024-04-24T14:35:38.985758+0000","last_fullsized":"2024-04-24T14:35:38.985758+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:13.379266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086492199999999996,"stat_sum":{"num_bytes":16777294,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":14,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":100663374,"num_keys_recovered":21,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"119'147","reported_seq":271,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:39.648487+0000","last_change":"2024-04-24T14:35:19.849974+0000","last_active":"2024-04-24T14:35:39.648487+0000","last_peered":"2024-04-24T14:35:39.648487+0000","last_clean":"2024-04-24T14:35:39.648487+0000","last_became_active":"2024-04-24T14:34:06.852598+0000","last_became_peered":"2024-04-24T14:34:06.852598+0000","last_unstale":"2024-04-24T14:35:39.648487+0000","last_undegraded":"2024-04-24T14:35:39.648487+0000","last_fullsized":"2024-04-24T14:35:39.648487+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:01:25.107060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00154912,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":10,"num_read":707,"num_read_kb":17181,"num_write":118,"num_write_kb":16433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777362,"num_keys_recovered":17,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"119'17","reported_seq":48,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.002380+0000","last_change":"2024-04-24T14:35:40.002380+0000","last_active":"2024-04-24T14:35:40.002380+0000","last_peered":"2024-04-24T14:35:40.002380+0000","last_clean":"2024-04-24T14:35:40.002380+0000","last_became_active":"2024-04-24T14:35:40.001994+0000","last_became_peered":"2024-04-24T14:35:40.001994+0000","last_unstale":"2024-04-24T14:35:40.002380+0000","last_undegraded":"2024-04-24T14:35:40.002380+0000","last_fullsized":"2024-04-24T14:35:40.002380+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:09.304847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061472399999999998,"stat_sum":{"num_bytes":20971636,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":16,"num_read_kb":135,"num_write":17,"num_write_kb":36868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":58720410,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"119'8","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981160+0000","last_change":"2024-04-24T14:35:19.843526+0000","last_active":"2024-04-24T14:35:38.981160+0000","last_peered":"2024-04-24T14:35:38.981160+0000","last_clean":"2024-04-24T14:35:38.981160+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:35:38.981160+0000","last_undegraded":"2024-04-24T14:35:38.981160+0000","last_fullsized":"2024-04-24T14:35:38.981160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001227351,"stat_sum":{"num_bytes":20971598,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":128,"num_write":6,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"119'7","reported_seq":149,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.000611+0000","last_change":"2024-04-24T14:35:40.000611+0000","last_active":"2024-04-24T14:35:40.000611+0000","last_peered":"2024-04-24T14:35:40.000611+0000","last_clean":"2024-04-24T14:35:40.000611+0000","last_became_active":"2024-04-24T14:35:40.000089+0000","last_became_peered":"2024-04-24T14:35:40.000089+0000","last_unstale":"2024-04-24T14:35:40.000611+0000","last_undegraded":"2024-04-24T14:35:40.000611+0000","last_fullsized":"2024-04-24T14:35:40.000611+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:48:12.340747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013719380000000001,"stat_sum":{"num_bytes":4194342,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165862,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"119'8","reported_seq":211,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981177+0000","last_change":"2024-04-24T14:35:19.842955+0000","last_active":"2024-04-24T14:35:38.981177+0000","last_peered":"2024-04-24T14:35:38.981177+0000","last_clean":"2024-04-24T14:35:38.981177+0000","last_became_active":"2024-04-24T14:34:06.852864+0000","last_became_peered":"2024-04-24T14:34:06.852864+0000","last_unstale":"2024-04-24T14:35:38.981177+0000","last_undegraded":"2024-04-24T14:35:38.981177+0000","last_fullsized":"2024-04-24T14:35:38.981177+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:42.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000833036,"stat_sum":{"num_bytes":114,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":8,"num_write_kb":8196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":46,"num_read":8174,"num_read_kb":182310,"num_write":895,"num_write_kb":139806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":189973067,"num_keys_recovered":73,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1083,"ondisk_log_size":1083,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":79692354,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":54,"num_read_kb":272,"num_write":69,"num_write_kb":131096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":297796236,"num_keys_recovered":70,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159473664,"data_stored":159384708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331744,"kb_used_data":30292,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040096,"statfs":{"total":96636764160,"available":96297058304,"internally_reserved":0,"allocated":31019008,"data_stored":30060920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79600000000000004,"15min":0.81299999999999994},"min":{"1min":0.61299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0529999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.54800000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.23,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.98699999999999999}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.82899999999999996}]},{"osd":2,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.83799999999999997},"min":{"1min":0.54100000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.026,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.87},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83899999999999997,"15min":0.85599999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0860000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.80200000000000005}]},{"osd":3,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79200000000000004,"15min":0.82099999999999995},"min":{"1min":0.52000000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.133,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.82299999999999995,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.99299999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.65800000000000003}]},{"osd":4,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.86099999999999999,"15min":0.873},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1259999999999999,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.873,"5min":0.93799999999999994,"15min":0.97899999999999998},"min":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.103,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.95599999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1662976,"data_stored":700809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.78000000000000003},"min":{"1min":0.49399999999999999,"5min":0.379,"15min":0.379},"max":{"1min":1.0800000000000001,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.60499999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0049999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.84799999999999998}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.95999999999999996}]},{"osd":2,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.83499999999999996,"15min":0.86099999999999999},"min":{"1min":0.63700000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.0189999999999999,"5min":1.113,"15min":1.113},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.64000000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0589999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.99199999999999999}]},{"osd":3,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83099999999999996,"15min":0.85099999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1970000000000001,"5min":1.931,"15min":1.931},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.85699999999999998,"15min":0.873},"min":{"1min":0.61499999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.161,"5min":1.591,"15min":1.591},"last":0.59099999999999997}]},{"osd":5,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.79300000000000004,"15min":0.78700000000000003},"min":{"1min":0.57999999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0629999999999999,"5min":1.087,"15min":1.087},"last":0.753},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.90800000000000003,"15min":0.92700000000000005},"min":{"1min":0.57999999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.87}]}]},{"osd":3,"up_from":11,"seq":47244640301,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328252,"kb_used_data":26792,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94043588,"statfs":{"total":96636764160,"available":96300634112,"internally_reserved":0,"allocated":27435008,"data_stored":26457195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.753,"15min":0.77300000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0009999999999999,"5min":1.282,"15min":1.282},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80300000000000005,"15min":0.80100000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.93300000000000005}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.86599999999999999}]},{"osd":2,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84999999999999998,"15min":0.872},"min":{"1min":0.64900000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.006,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.91900000000000004,"15min":0.93600000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.238,"5min":1.337,"15min":1.337},"last":1.0389999999999999}]},{"osd":4,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.63200000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.52000000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.748}]},{"osd":5,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79400000000000004,"15min":0.78800000000000003},"min":{"1min":0.625,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1200000000000001,"5min":1.367,"15min":1.367},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.871,"15min":0.872},"min":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.4199999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.98899999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319456,"kb_used_data":18004,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94052384,"statfs":{"total":96636764160,"available":96309641216,"internally_reserved":0,"allocated":18436096,"data_stored":17477987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85899999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60599999999999998},"min":{"1min":0.36299999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.871,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.67100000000000004}]},{"osd":1,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.71799999999999997}]},{"osd":3,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.83899999999999997,"15min":0.85299999999999998},"min":{"1min":0.64500000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.117,"5min":1.147,"15min":1.147},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.873,"15min":0.90500000000000003},"min":{"1min":0.64200000000000002,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.98799999999999999,"5min":1.77,"15min":1.77},"last":0.94899999999999995}]},{"osd":4,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.85599999999999998,"15min":0.85699999999999998},"min":{"1min":0.70799999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.88400000000000001,"15min":0.89600000000000002},"min":{"1min":0.76500000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":0.88500000000000001}]},{"osd":5,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91900000000000004,"15min":0.95499999999999996},"min":{"1min":0.623,"5min":0.62,"15min":0.62},"max":{"1min":1.123,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.91800000000000004,"15min":0.92600000000000005},"min":{"1min":0.77500000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.97799999999999998}]}]},{"osd":1,"up_from":66,"seq":283467841559,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344696,"kb_used_data":46700,"kb_used_omap":4,"kb_used_meta":297979,"kb_avail":94027144,"statfs":{"total":96636764160,"available":96283795456,"internally_reserved":0,"allocated":47820800,"data_stored":46838444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305131073},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":0.54500000000000004}]},{"osd":2,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":0.60299999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.79800000000000004}]},{"osd":4,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.86199999999999999}]},{"osd":5,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.99399999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640302,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348716,"kb_used_data":43164,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94023124,"statfs":{"total":96636764160,"available":96279678976,"internally_reserved":0,"allocated":44199936,"data_stored":43234297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 14:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.41999999999999998}]},{"osd":2,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.316,"15min":0.316},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68000000000000005,"15min":0.67900000000000005},"min":{"1min":0.45800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.70299999999999996}]},{"osd":3,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.022,"5min":1.472,"15min":1.472},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81100000000000005,"15min":0.80700000000000005},"min":{"1min":0.51600000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1299999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.65300000000000002}]},{"osd":4,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.89400000000000002,"15min":0.91700000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0309999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.91800000000000004,"15min":0.91900000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2210000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.77900000000000003}]},{"osd":5,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.70299999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.256,"5min":1.641,"15min":1.641},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.92200000000000004,"15min":0.92800000000000005},"min":{"1min":0.66900000000000004,"5min":0.628,"15min":0.628},"max":{"1min":1.1359999999999999,"5min":1.534,"15min":1.534},"last":0.752}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46137730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16785408,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25190400,"data_stored":25166132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:35:48.912 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T14:35:48.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:35:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:49.200 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:35:49.211 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-04-24T14:35:47.227174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80282741,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":46,"num_read":8344,"num_read_kb":182682,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":487769303,"num_keys_recovered":143,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1267,"ondisk_log_size":1267,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1975948,"kb_used_data":166576,"kb_used_omap":12,"kb_used_meta":1809267,"kb_avail":564255092,"statfs":{"total":579820584960,"available":577797214208,"internally_reserved":0,"allocated":170573824,"data_stored":164769652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13063,"internal_metadata":1852689657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-28,"num_bytes_recovered":16777275,"num_keys_recovered":2,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.248732"},"pg_stats":[{"pgid":"1.0","version":"14'117","reported_seq":281,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981208+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:35:38.981208+0000","last_peered":"2024-04-24T14:35:38.981208+0000","last_clean":"2024-04-24T14:35:38.981208+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:35:38.981208+0000","last_undegraded":"2024-04-24T14:35:38.981208+0000","last_fullsized":"2024-04-24T14:35:38.981208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"49'3","reported_seq":68,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.984346+0000","last_change":"2024-04-24T14:35:19.912506+0000","last_active":"2024-04-24T14:35:38.984346+0000","last_peered":"2024-04-24T14:35:38.984346+0000","last_clean":"2024-04-24T14:35:38.984346+0000","last_became_active":"2024-04-24T14:35:04.316770+0000","last_became_peered":"2024-04-24T14:35:04.316770+0000","last_unstale":"2024-04-24T14:35:38.984346+0000","last_undegraded":"2024-04-24T14:35:38.984346+0000","last_fullsized":"2024-04-24T14:35:38.984346+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:06.338365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058253000000000005,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"119'187","reported_seq":617,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.003333+0000","last_change":"2024-04-24T14:35:40.003333+0000","last_active":"2024-04-24T14:35:40.003333+0000","last_peered":"2024-04-24T14:35:40.003333+0000","last_clean":"2024-04-24T14:35:40.003333+0000","last_became_active":"2024-04-24T14:35:40.002805+0000","last_became_peered":"2024-04-24T14:35:40.002805+0000","last_unstale":"2024-04-24T14:35:40.003333+0000","last_undegraded":"2024-04-24T14:35:40.003333+0000","last_fullsized":"2024-04-24T14:35:40.003333+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:02.689379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013315029999999999,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":14,"num_read":1693,"num_read_kb":48628,"num_write":128,"num_write_kb":32867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":104988999,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"119'9","reported_seq":229,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981136+0000","last_change":"2024-04-24T14:35:28.058638+0000","last_active":"2024-04-24T14:35:38.981136+0000","last_peered":"2024-04-24T14:35:38.981136+0000","last_clean":"2024-04-24T14:35:38.981136+0000","last_became_active":"2024-04-24T14:35:27.881143+0000","last_became_peered":"2024-04-24T14:35:27.881143+0000","last_unstale":"2024-04-24T14:35:38.981136+0000","last_undegraded":"2024-04-24T14:35:38.981136+0000","last_fullsized":"2024-04-24T14:35:38.981136+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:08.850822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076176800000000002,"stat_sum":{"num_bytes":78,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":8,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165902,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"117'453","reported_seq":3073,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.982741+0000","last_change":"2024-04-24T14:35:29.912112+0000","last_active":"2024-04-24T14:35:38.982741+0000","last_peered":"2024-04-24T14:35:38.982741+0000","last_clean":"2024-04-24T14:35:38.982741+0000","last_became_active":"2024-04-24T14:35:29.910973+0000","last_became_peered":"2024-04-24T14:35:29.910973+0000","last_unstale":"2024-04-24T14:35:38.982741+0000","last_undegraded":"2024-04-24T14:35:38.982741+0000","last_fullsized":"2024-04-24T14:35:38.982741+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:47:08.597146+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0051668080000000002,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":11,"num_read":4739,"num_read_kb":77733,"num_write":290,"num_write_kb":56447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":25165862,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"119'7","reported_seq":235,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.988225+0000","last_change":"2024-04-24T14:35:27.935655+0000","last_active":"2024-04-24T14:35:38.988225+0000","last_peered":"2024-04-24T14:35:38.988225+0000","last_clean":"2024-04-24T14:35:38.988225+0000","last_became_active":"2024-04-24T14:35:27.884423+0000","last_became_peered":"2024-04-24T14:35:27.884423+0000","last_unstale":"2024-04-24T14:35:38.988225+0000","last_undegraded":"2024-04-24T14:35:38.988225+0000","last_fullsized":"2024-04-24T14:35:38.988225+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:30.623924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085674600000000003,"stat_sum":{"num_bytes":4194380,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":6,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":20971710,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"119'296","reported_seq":829,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.007103+0000","last_change":"2024-04-24T14:35:40.007103+0000","last_active":"2024-04-24T14:35:40.007103+0000","last_peered":"2024-04-24T14:35:40.007103+0000","last_clean":"2024-04-24T14:35:40.007103+0000","last_became_active":"2024-04-24T14:35:40.006335+0000","last_became_peered":"2024-04-24T14:35:40.006335+0000","last_unstale":"2024-04-24T14:35:40.007103+0000","last_undegraded":"2024-04-24T14:35:40.007103+0000","last_fullsized":"2024-04-24T14:35:40.007103+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:49:50.086262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014897840000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1035,"num_read_kb":38768,"num_write":359,"num_write_kb":34059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":43040844,"num_keys_recovered":6,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"119'8","reported_seq":103,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.985758+0000","last_change":"2024-04-24T14:35:28.046471+0000","last_active":"2024-04-24T14:35:38.985758+0000","last_peered":"2024-04-24T14:35:38.985758+0000","last_clean":"2024-04-24T14:35:38.985758+0000","last_became_active":"2024-04-24T14:35:27.885728+0000","last_became_peered":"2024-04-24T14:35:27.885728+0000","last_unstale":"2024-04-24T14:35:38.985758+0000","last_undegraded":"2024-04-24T14:35:38.985758+0000","last_fullsized":"2024-04-24T14:35:38.985758+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:13.379266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086492199999999996,"stat_sum":{"num_bytes":16777294,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":14,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":100663374,"num_keys_recovered":21,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"119'147","reported_seq":271,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:39.648487+0000","last_change":"2024-04-24T14:35:19.849974+0000","last_active":"2024-04-24T14:35:39.648487+0000","last_peered":"2024-04-24T14:35:39.648487+0000","last_clean":"2024-04-24T14:35:39.648487+0000","last_became_active":"2024-04-24T14:34:06.852598+0000","last_became_peered":"2024-04-24T14:34:06.852598+0000","last_unstale":"2024-04-24T14:35:39.648487+0000","last_undegraded":"2024-04-24T14:35:39.648487+0000","last_fullsized":"2024-04-24T14:35:39.648487+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:01:25.107060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00154912,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":10,"num_read":707,"num_read_kb":17181,"num_write":118,"num_write_kb":16433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777362,"num_keys_recovered":17,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"119'17","reported_seq":48,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.002380+0000","last_change":"2024-04-24T14:35:40.002380+0000","last_active":"2024-04-24T14:35:40.002380+0000","last_peered":"2024-04-24T14:35:40.002380+0000","last_clean":"2024-04-24T14:35:40.002380+0000","last_became_active":"2024-04-24T14:35:40.001994+0000","last_became_peered":"2024-04-24T14:35:40.001994+0000","last_unstale":"2024-04-24T14:35:40.002380+0000","last_undegraded":"2024-04-24T14:35:40.002380+0000","last_fullsized":"2024-04-24T14:35:40.002380+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:09.304847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061472399999999998,"stat_sum":{"num_bytes":20971636,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":16,"num_read_kb":135,"num_write":17,"num_write_kb":36868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":58720410,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"119'8","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981160+0000","last_change":"2024-04-24T14:35:19.843526+0000","last_active":"2024-04-24T14:35:38.981160+0000","last_peered":"2024-04-24T14:35:38.981160+0000","last_clean":"2024-04-24T14:35:38.981160+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:35:38.981160+0000","last_undegraded":"2024-04-24T14:35:38.981160+0000","last_fullsized":"2024-04-24T14:35:38.981160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001227351,"stat_sum":{"num_bytes":20971598,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":128,"num_write":6,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"119'7","reported_seq":149,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.000611+0000","last_change":"2024-04-24T14:35:40.000611+0000","last_active":"2024-04-24T14:35:40.000611+0000","last_peered":"2024-04-24T14:35:40.000611+0000","last_clean":"2024-04-24T14:35:40.000611+0000","last_became_active":"2024-04-24T14:35:40.000089+0000","last_became_peered":"2024-04-24T14:35:40.000089+0000","last_unstale":"2024-04-24T14:35:40.000611+0000","last_undegraded":"2024-04-24T14:35:40.000611+0000","last_fullsized":"2024-04-24T14:35:40.000611+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:48:12.340747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013719380000000001,"stat_sum":{"num_bytes":4194342,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165862,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"119'8","reported_seq":211,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981177+0000","last_change":"2024-04-24T14:35:19.842955+0000","last_active":"2024-04-24T14:35:38.981177+0000","last_peered":"2024-04-24T14:35:38.981177+0000","last_clean":"2024-04-24T14:35:38.981177+0000","last_became_active":"2024-04-24T14:34:06.852864+0000","last_became_peered":"2024-04-24T14:34:06.852864+0000","last_unstale":"2024-04-24T14:35:38.981177+0000","last_undegraded":"2024-04-24T14:35:38.981177+0000","last_fullsized":"2024-04-24T14:35:38.981177+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:42.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000833036,"stat_sum":{"num_bytes":114,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":8,"num_write_kb":8196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":46,"num_read":8174,"num_read_kb":182310,"num_write":895,"num_write_kb":139806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":189973067,"num_keys_recovered":73,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1083,"ondisk_log_size":1083,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":79692354,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":54,"num_read_kb":272,"num_write":69,"num_write_kb":131096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":297796236,"num_keys_recovered":70,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159473664,"data_stored":159384708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331744,"kb_used_data":30292,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040096,"statfs":{"total":96636764160,"available":96297058304,"internally_reserved":0,"allocated":31019008,"data_stored":30060920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79600000000000004,"15min":0.81299999999999994},"min":{"1min":0.61299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0529999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.54800000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.23,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.98699999999999999}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.82899999999999996}]},{"osd":2,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.83799999999999997},"min":{"1min":0.54100000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.026,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.87},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83899999999999997,"15min":0.85599999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0860000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.80200000000000005}]},{"osd":3,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79200000000000004,"15min":0.82099999999999995},"min":{"1min":0.52000000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.133,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.82299999999999995,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.99299999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.65800000000000003}]},{"osd":4,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.86099999999999999,"15min":0.873},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1259999999999999,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.873,"5min":0.93799999999999994,"15min":0.97899999999999998},"min":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.103,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.95599999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1662976,"data_stored":700809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.78000000000000003},"min":{"1min":0.49399999999999999,"5min":0.379,"15min":0.379},"max":{"1min":1.0800000000000001,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.60499999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0049999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.84799999999999998}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.95999999999999996}]},{"osd":2,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.83499999999999996,"15min":0.86099999999999999},"min":{"1min":0.63700000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.0189999999999999,"5min":1.113,"15min":1.113},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.64000000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0589999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.99199999999999999}]},{"osd":3,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83099999999999996,"15min":0.85099999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1970000000000001,"5min":1.931,"15min":1.931},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.85699999999999998,"15min":0.873},"min":{"1min":0.61499999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.161,"5min":1.591,"15min":1.591},"last":0.59099999999999997}]},{"osd":5,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.79300000000000004,"15min":0.78700000000000003},"min":{"1min":0.57999999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0629999999999999,"5min":1.087,"15min":1.087},"last":0.753},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.90800000000000003,"15min":0.92700000000000005},"min":{"1min":0.57999999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.87}]}]},{"osd":3,"up_from":11,"seq":47244640301,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328252,"kb_used_data":26792,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94043588,"statfs":{"total":96636764160,"available":96300634112,"internally_reserved":0,"allocated":27435008,"data_stored":26457195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.753,"15min":0.77300000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0009999999999999,"5min":1.282,"15min":1.282},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80300000000000005,"15min":0.80100000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.93300000000000005}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.86599999999999999}]},{"osd":2,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84999999999999998,"15min":0.872},"min":{"1min":0.64900000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.006,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.91900000000000004,"15min":0.93600000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.238,"5min":1.337,"15min":1.337},"last":1.0389999999999999}]},{"osd":4,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.63200000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.52000000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.748}]},{"osd":5,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79400000000000004,"15min":0.78800000000000003},"min":{"1min":0.625,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1200000000000001,"5min":1.367,"15min":1.367},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.871,"15min":0.872},"min":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.4199999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.98899999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319456,"kb_used_data":18004,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94052384,"statfs":{"total":96636764160,"available":96309641216,"internally_reserved":0,"allocated":18436096,"data_stored":17477987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85899999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60599999999999998},"min":{"1min":0.36299999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.871,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.67100000000000004}]},{"osd":1,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.71799999999999997}]},{"osd":3,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.83899999999999997,"15min":0.85299999999999998},"min":{"1min":0.64500000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.117,"5min":1.147,"15min":1.147},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.873,"15min":0.90500000000000003},"min":{"1min":0.64200000000000002,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.98799999999999999,"5min":1.77,"15min":1.77},"last":0.94899999999999995}]},{"osd":4,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.85599999999999998,"15min":0.85699999999999998},"min":{"1min":0.70799999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.88400000000000001,"15min":0.89600000000000002},"min":{"1min":0.76500000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":0.88500000000000001}]},{"osd":5,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91900000000000004,"15min":0.95499999999999996},"min":{"1min":0.623,"5min":0.62,"15min":0.62},"max":{"1min":1.123,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.91800000000000004,"15min":0.92600000000000005},"min":{"1min":0.77500000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.97799999999999998}]}]},{"osd":1,"up_from":66,"seq":283467841559,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344696,"kb_used_data":46700,"kb_used_omap":4,"kb_used_meta":297979,"kb_avail":94027144,"statfs":{"total":96636764160,"available":96283795456,"internally_reserved":0,"allocated":47820800,"data_stored":46838444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305131073},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":0.54500000000000004}]},{"osd":2,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":0.60299999999999998}]},{"osd":3,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.79800000000000004}]},{"osd":4,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.86199999999999999}]},{"osd":5,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.99399999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640302,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348716,"kb_used_data":43164,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94023124,"statfs":{"total":96636764160,"available":96279678976,"internally_reserved":0,"allocated":44199936,"data_stored":43234297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 14:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.41999999999999998}]},{"osd":2,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.316,"15min":0.316},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68000000000000005,"15min":0.67900000000000005},"min":{"1min":0.45800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.70299999999999996}]},{"osd":3,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.022,"5min":1.472,"15min":1.472},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81100000000000005,"15min":0.80700000000000005},"min":{"1min":0.51600000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1299999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.65300000000000002}]},{"osd":4,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.89400000000000002,"15min":0.91700000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0309999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.91800000000000004,"15min":0.91900000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2210000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.77900000000000003}]},{"osd":5,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.70299999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.256,"5min":1.641,"15min":1.641},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.92200000000000004,"15min":0.92800000000000005},"min":{"1min":0.66900000000000004,"5min":0.628,"15min":0.628},"max":{"1min":1.1359999999999999,"5min":1.534,"15min":1.534},"last":0.752}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46137730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16785408,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25190400,"data_stored":25166132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:35:49.212 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T14:35:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:49.496 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":138,"fsid":"6706935e-d09a-472b-b839-321319fa9e54","created":"2024-04-24T14:32:09.059925+0000","modified":"2024-04-24T14:35:39.977401+0000","last_up_change":"2024-04-24T14:34:00.856741+0000","last_in_change":"2024-04-24T14:35:26.859498+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T14:32:21.233022+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T14:32:26.609964+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"21","last_force_op_resend_prenautilus":"21","last_force_op_resend_preluminous":"21","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":119,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[3],"tier_of":-1,"read_tier":3,"write_tier":3,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cache","create_time":"2024-04-24T14:32:33.987564+0000","flags":8201,"flags_names":"hashpspool,incomplete_clones,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"21","last_force_op_resend_prenautilus":"21","last_force_op_resend_preluminous":"21","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":119,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":2,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a2506b4f-3486-4f51-972f-1d9e4b015346","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":20309},{"type":"v1","addr":"172.21.15.90:6817","nonce":20309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":20309},{"type":"v1","addr":"172.21.15.90:6819","nonce":20309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":20309},{"type":"v1","addr":"172.21.15.90:6823","nonce":20309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":20309},{"type":"v1","addr":"172.21.15.90:6821","nonce":20309}]},"public_addr":"172.21.15.90:6817/20309","cluster_addr":"172.21.15.90:6819/20309","heartbeat_back_addr":"172.21.15.90:6823/20309","heartbeat_front_addr":"172.21.15.90:6821/20309","state":["exists","up"]},{"osd":1,"uuid":"9e17324f-a615-494f-ba0d-7c8aa8dac95c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":31,"up_from":66,"up_thru":137,"down_at":32,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":34413},{"type":"v1","addr":"172.21.15.90:6801","nonce":34413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":34413},{"type":"v1","addr":"172.21.15.90:6803","nonce":34413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":34413},{"type":"v1","addr":"172.21.15.90:6807","nonce":34413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":34413},{"type":"v1","addr":"172.21.15.90:6805","nonce":34413}]},"public_addr":"172.21.15.90:6801/34413","cluster_addr":"172.21.15.90:6803/34413","heartbeat_back_addr":"172.21.15.90:6807/34413","heartbeat_front_addr":"172.21.15.90:6805/34413","state":["exists","up"]},{"osd":2,"uuid":"f7fb40c8-bdad-442c-bf15-7b3e5b3ed5e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":20310},{"type":"v1","addr":"172.21.15.90:6809","nonce":20310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":20310},{"type":"v1","addr":"172.21.15.90:6811","nonce":20310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":20310},{"type":"v1","addr":"172.21.15.90:6815","nonce":20310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":20310},{"type":"v1","addr":"172.21.15.90:6813","nonce":20310}]},"public_addr":"172.21.15.90:6809/20310","cluster_addr":"172.21.15.90:6811/20310","heartbeat_back_addr":"172.21.15.90:6815/20310","heartbeat_front_addr":"172.21.15.90:6813/20310","state":["exists","up"]},{"osd":3,"uuid":"509c16da-22b2-48e8-aa51-c320c8c4fd8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":20316},{"type":"v1","addr":"172.21.15.103:6801","nonce":20316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":20316},{"type":"v1","addr":"172.21.15.103:6803","nonce":20316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":20316},{"type":"v1","addr":"172.21.15.103:6807","nonce":20316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":20316},{"type":"v1","addr":"172.21.15.103:6805","nonce":20316}]},"public_addr":"172.21.15.103:6801/20316","cluster_addr":"172.21.15.103:6803/20316","heartbeat_back_addr":"172.21.15.103:6807/20316","heartbeat_front_addr":"172.21.15.103:6805/20316","state":["exists","up"]},{"osd":4,"uuid":"1462066b-4e59-46d4-9b85-1f61ee1d56a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":20318},{"type":"v1","addr":"172.21.15.103:6809","nonce":20318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":20318},{"type":"v1","addr":"172.21.15.103:6811","nonce":20318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":20318},{"type":"v1","addr":"172.21.15.103:6815","nonce":20318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":20318},{"type":"v1","addr":"172.21.15.103:6813","nonce":20318}]},"public_addr":"172.21.15.103:6809/20318","cluster_addr":"172.21.15.103:6811/20318","heartbeat_back_addr":"172.21.15.103:6815/20318","heartbeat_front_addr":"172.21.15.103:6813/20318","state":["exists","up"]},{"osd":5,"uuid":"4dc954a8-5f3c-4826-b9bc-98cf30519382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":20317},{"type":"v1","addr":"172.21.15.103:6817","nonce":20317}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":20317},{"type":"v1","addr":"172.21.15.103:6819","nonce":20317}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":20317},{"type":"v1","addr":"172.21.15.103:6823","nonce":20317}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":20317},{"type":"v1","addr":"172.21.15.103:6821","nonce":20317}]},"public_addr":"172.21.15.103:6817/20317","cluster_addr":"172.21.15.103:6819/20317","heartbeat_back_addr":"172.21.15.103:6823/20317","heartbeat_front_addr":"172.21.15.103:6821/20317","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.792899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.580414+0000","dead_epoch":32},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.713336+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.579638+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.570510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T14:32:16.736558+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T14:35:50.510 INFO:tasks.ceph:Scrubbing osd.0 2024-04-24T14:35:50.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-24T14:35:50.663 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:50.663 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T14:35:50.663 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:50.676 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-24T14:35:50.975 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 0 to deep-scrub 2024-04-24T14:35:50.985 INFO:tasks.ceph:Scrubbing osd.1 2024-04-24T14:35:50.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-24T14:35:51.124 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:51.125 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T14:35:51.125 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:51.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-24T14:35:51.443 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 1 to deep-scrub 2024-04-24T14:35:51.457 INFO:tasks.ceph:Scrubbing osd.2 2024-04-24T14:35:51.457 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-24T14:35:51.602 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:51.602 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T14:35:51.602 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:51.612 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-24T14:35:51.908 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 2 to deep-scrub 2024-04-24T14:35:51.921 INFO:tasks.ceph:Scrubbing osd.3 2024-04-24T14:35:51.921 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-24T14:35:52.070 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:52.071 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T14:35:52.071 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:52.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-24T14:35:52.381 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 3 to deep-scrub 2024-04-24T14:35:52.391 INFO:tasks.ceph:Scrubbing osd.4 2024-04-24T14:35:52.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-24T14:35:52.541 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:52.541 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T14:35:52.542 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:52.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-24T14:35:52.854 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 4 to deep-scrub 2024-04-24T14:35:52.864 INFO:tasks.ceph:Scrubbing osd.5 2024-04-24T14:35:52.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-24T14:35:52.999 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-04-24T14:35:52.999 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T14:35:52.999 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-04-24T14:35:53.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-24T14:35:53.301 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 5 to deep-scrub 2024-04-24T14:35:53.311 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:35:53.593 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:35:53.596 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:35:53.608 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-04-24T14:35:53.228890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80282741,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":46,"num_read":8344,"num_read_kb":182682,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":487769303,"num_keys_recovered":143,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1267,"ondisk_log_size":1267,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1975948,"kb_used_data":166576,"kb_used_omap":12,"kb_used_meta":1809267,"kb_avail":564255092,"statfs":{"total":579820584960,"available":577797214208,"internally_reserved":0,"allocated":170573824,"data_stored":164769652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13063,"internal_metadata":1852689657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003537"},"pg_stats":[{"pgid":"1.0","version":"14'117","reported_seq":281,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981208+0000","last_change":"2024-04-24T14:32:22.272546+0000","last_active":"2024-04-24T14:35:38.981208+0000","last_peered":"2024-04-24T14:35:38.981208+0000","last_clean":"2024-04-24T14:35:38.981208+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:35:38.981208+0000","last_undegraded":"2024-04-24T14:35:38.981208+0000","last_fullsized":"2024-04-24T14:35:38.981208+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:21.251204+0000","last_clean_scrub_stamp":"2024-04-24T14:32:21.251204+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.529841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"49'3","reported_seq":68,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.984346+0000","last_change":"2024-04-24T14:35:19.912506+0000","last_active":"2024-04-24T14:35:38.984346+0000","last_peered":"2024-04-24T14:35:38.984346+0000","last_clean":"2024-04-24T14:35:38.984346+0000","last_became_active":"2024-04-24T14:35:04.316770+0000","last_became_peered":"2024-04-24T14:35:04.316770+0000","last_unstale":"2024-04-24T14:35:38.984346+0000","last_undegraded":"2024-04-24T14:35:38.984346+0000","last_fullsized":"2024-04-24T14:35:38.984346+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:14:06.338365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058253000000000005,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"119'187","reported_seq":617,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.003333+0000","last_change":"2024-04-24T14:35:40.003333+0000","last_active":"2024-04-24T14:35:40.003333+0000","last_peered":"2024-04-24T14:35:40.003333+0000","last_clean":"2024-04-24T14:35:40.003333+0000","last_became_active":"2024-04-24T14:35:40.002805+0000","last_became_peered":"2024-04-24T14:35:40.002805+0000","last_unstale":"2024-04-24T14:35:40.003333+0000","last_undegraded":"2024-04-24T14:35:40.003333+0000","last_fullsized":"2024-04-24T14:35:40.003333+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:02.689379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013315029999999999,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":14,"num_read":1693,"num_read_kb":48628,"num_write":128,"num_write_kb":32867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":104988999,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"119'9","reported_seq":229,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981136+0000","last_change":"2024-04-24T14:35:28.058638+0000","last_active":"2024-04-24T14:35:38.981136+0000","last_peered":"2024-04-24T14:35:38.981136+0000","last_clean":"2024-04-24T14:35:38.981136+0000","last_became_active":"2024-04-24T14:35:27.881143+0000","last_became_peered":"2024-04-24T14:35:27.881143+0000","last_unstale":"2024-04-24T14:35:38.981136+0000","last_undegraded":"2024-04-24T14:35:38.981136+0000","last_fullsized":"2024-04-24T14:35:38.981136+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:21:08.850822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076176800000000002,"stat_sum":{"num_bytes":78,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":8,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165902,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"117'453","reported_seq":3074,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:39.999616+0000","last_change":"2024-04-24T14:35:29.912112+0000","last_active":"2024-04-24T14:35:39.999616+0000","last_peered":"2024-04-24T14:35:39.999616+0000","last_clean":"2024-04-24T14:35:39.999616+0000","last_became_active":"2024-04-24T14:35:29.910973+0000","last_became_peered":"2024-04-24T14:35:29.910973+0000","last_unstale":"2024-04-24T14:35:39.999616+0000","last_undegraded":"2024-04-24T14:35:39.999616+0000","last_fullsized":"2024-04-24T14:35:39.999616+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0051668080000000002,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":11,"num_read":4739,"num_read_kb":77733,"num_write":290,"num_write_kb":56447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":25165862,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"119'7","reported_seq":235,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.988225+0000","last_change":"2024-04-24T14:35:27.935655+0000","last_active":"2024-04-24T14:35:38.988225+0000","last_peered":"2024-04-24T14:35:38.988225+0000","last_clean":"2024-04-24T14:35:38.988225+0000","last_became_active":"2024-04-24T14:35:27.884423+0000","last_became_peered":"2024-04-24T14:35:27.884423+0000","last_unstale":"2024-04-24T14:35:38.988225+0000","last_undegraded":"2024-04-24T14:35:38.988225+0000","last_fullsized":"2024-04-24T14:35:38.988225+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:30.623924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085674600000000003,"stat_sum":{"num_bytes":4194380,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":6,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":20971710,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"119'296","reported_seq":829,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.007103+0000","last_change":"2024-04-24T14:35:40.007103+0000","last_active":"2024-04-24T14:35:40.007103+0000","last_peered":"2024-04-24T14:35:40.007103+0000","last_clean":"2024-04-24T14:35:40.007103+0000","last_became_active":"2024-04-24T14:35:40.006335+0000","last_became_peered":"2024-04-24T14:35:40.006335+0000","last_unstale":"2024-04-24T14:35:40.007103+0000","last_undegraded":"2024-04-24T14:35:40.007103+0000","last_fullsized":"2024-04-24T14:35:40.007103+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:49:50.086262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014897840000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1035,"num_read_kb":38768,"num_write":359,"num_write_kb":34059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":43040844,"num_keys_recovered":6,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"119'8","reported_seq":103,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.985758+0000","last_change":"2024-04-24T14:35:28.046471+0000","last_active":"2024-04-24T14:35:38.985758+0000","last_peered":"2024-04-24T14:35:38.985758+0000","last_clean":"2024-04-24T14:35:38.985758+0000","last_became_active":"2024-04-24T14:35:27.885728+0000","last_became_peered":"2024-04-24T14:35:27.885728+0000","last_unstale":"2024-04-24T14:35:38.985758+0000","last_undegraded":"2024-04-24T14:35:38.985758+0000","last_fullsized":"2024-04-24T14:35:38.985758+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:46:13.379266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086492199999999996,"stat_sum":{"num_bytes":16777294,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":14,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":100663374,"num_keys_recovered":21,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"119'147","reported_seq":271,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:39.648487+0000","last_change":"2024-04-24T14:35:19.849974+0000","last_active":"2024-04-24T14:35:39.648487+0000","last_peered":"2024-04-24T14:35:39.648487+0000","last_clean":"2024-04-24T14:35:39.648487+0000","last_became_active":"2024-04-24T14:34:06.852598+0000","last_became_peered":"2024-04-24T14:34:06.852598+0000","last_unstale":"2024-04-24T14:35:39.648487+0000","last_undegraded":"2024-04-24T14:35:39.648487+0000","last_fullsized":"2024-04-24T14:35:39.648487+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:34.325783+0000","last_clean_scrub_stamp":"2024-04-24T14:32:34.325783+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:01:25.107060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00154912,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":10,"num_read":707,"num_read_kb":17181,"num_write":118,"num_write_kb":16433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777362,"num_keys_recovered":17,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"119'17","reported_seq":48,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.002380+0000","last_change":"2024-04-24T14:35:40.002380+0000","last_active":"2024-04-24T14:35:40.002380+0000","last_peered":"2024-04-24T14:35:40.002380+0000","last_clean":"2024-04-24T14:35:40.002380+0000","last_became_active":"2024-04-24T14:35:40.001994+0000","last_became_peered":"2024-04-24T14:35:40.001994+0000","last_unstale":"2024-04-24T14:35:40.002380+0000","last_undegraded":"2024-04-24T14:35:40.002380+0000","last_fullsized":"2024-04-24T14:35:40.002380+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:09.304847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061472399999999998,"stat_sum":{"num_bytes":20971636,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":16,"num_read_kb":135,"num_write":17,"num_write_kb":36868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":58720410,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"119'8","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981160+0000","last_change":"2024-04-24T14:35:19.843526+0000","last_active":"2024-04-24T14:35:38.981160+0000","last_peered":"2024-04-24T14:35:38.981160+0000","last_clean":"2024-04-24T14:35:38.981160+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:35:38.981160+0000","last_undegraded":"2024-04-24T14:35:38.981160+0000","last_fullsized":"2024-04-24T14:35:38.981160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:11.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001227351,"stat_sum":{"num_bytes":20971598,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":128,"num_write":6,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"119'7","reported_seq":149,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:40.000611+0000","last_change":"2024-04-24T14:35:40.000611+0000","last_active":"2024-04-24T14:35:40.000611+0000","last_peered":"2024-04-24T14:35:40.000611+0000","last_clean":"2024-04-24T14:35:40.000611+0000","last_became_active":"2024-04-24T14:35:40.000089+0000","last_became_peered":"2024-04-24T14:35:40.000089+0000","last_unstale":"2024-04-24T14:35:40.000611+0000","last_undegraded":"2024-04-24T14:35:40.000611+0000","last_fullsized":"2024-04-24T14:35:40.000611+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:48:12.340747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013719380000000001,"stat_sum":{"num_bytes":4194342,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165862,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"119'8","reported_seq":211,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-04-24T14:35:38.981177+0000","last_change":"2024-04-24T14:35:19.842955+0000","last_active":"2024-04-24T14:35:38.981177+0000","last_peered":"2024-04-24T14:35:38.981177+0000","last_clean":"2024-04-24T14:35:38.981177+0000","last_became_active":"2024-04-24T14:34:06.852864+0000","last_became_peered":"2024-04-24T14:34:06.852864+0000","last_unstale":"2024-04-24T14:35:38.981177+0000","last_undegraded":"2024-04-24T14:35:38.981177+0000","last_fullsized":"2024-04-24T14:35:38.981177+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T14:32:27.283594+0000","last_clean_scrub_stamp":"2024-04-24T14:32:27.283594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:42.853065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000833036,"stat_sum":{"num_bytes":114,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":8,"num_write_kb":8196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":46,"num_read":8174,"num_read_kb":182310,"num_write":895,"num_write_kb":139806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":189973067,"num_keys_recovered":73,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1083,"ondisk_log_size":1083,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":79692354,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":54,"num_read_kb":272,"num_write":69,"num_write_kb":131096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":297796236,"num_keys_recovered":70,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159473664,"data_stored":159384708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640304,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331744,"kb_used_data":30292,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040096,"statfs":{"total":96636764160,"available":96297058304,"internally_reserved":0,"allocated":31019008,"data_stored":30060920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79600000000000004,"15min":0.81299999999999994},"min":{"1min":0.61299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0529999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.54800000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.23,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.72099999999999997}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.79100000000000004}]},{"osd":2,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.83799999999999997},"min":{"1min":0.54100000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.026,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83899999999999997,"15min":0.85599999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0860000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.81599999999999995}]},{"osd":3,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79200000000000004,"15min":0.82099999999999995},"min":{"1min":0.52000000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.133,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.82299999999999995,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.99299999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.76100000000000001}]},{"osd":4,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.86099999999999999,"15min":0.873},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1259999999999999,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.873,"5min":0.93799999999999994,"15min":0.97899999999999998},"min":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.103,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.65600000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640304,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303084,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94068756,"statfs":{"total":96636764160,"available":96326406144,"internally_reserved":0,"allocated":1662976,"data_stored":700809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.78000000000000003},"min":{"1min":0.49399999999999999,"5min":0.379,"15min":0.379},"max":{"1min":1.0800000000000001,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.60499999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0049999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.65200000000000002}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.73099999999999998}]},{"osd":2,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.83499999999999996,"15min":0.86099999999999999},"min":{"1min":0.63700000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.0189999999999999,"5min":1.113,"15min":1.113},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.64000000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0589999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.91700000000000004}]},{"osd":3,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83099999999999996,"15min":0.85099999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1970000000000001,"5min":1.931,"15min":1.931},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.85699999999999998,"15min":0.873},"min":{"1min":0.61499999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.161,"5min":1.591,"15min":1.591},"last":0.98399999999999999}]},{"osd":5,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.79300000000000004,"15min":0.78700000000000003},"min":{"1min":0.57999999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0629999999999999,"5min":1.087,"15min":1.087},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.90800000000000003,"15min":0.92700000000000005},"min":{"1min":0.57999999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.81599999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640302,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328252,"kb_used_data":26792,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94043588,"statfs":{"total":96636764160,"available":96300634112,"internally_reserved":0,"allocated":27435008,"data_stored":26457195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.753,"15min":0.77300000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0009999999999999,"5min":1.282,"15min":1.282},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80300000000000005,"15min":0.80100000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.85899999999999999}]},{"osd":1,"last update":"Wed Apr 24 14:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.61699999999999999}]},{"osd":2,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84999999999999998,"15min":0.872},"min":{"1min":0.64900000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.006,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.91900000000000004,"15min":0.93600000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.238,"5min":1.337,"15min":1.337},"last":0.56200000000000006}]},{"osd":4,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.63200000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.52000000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.71799999999999997}]},{"osd":5,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79400000000000004,"15min":0.78800000000000003},"min":{"1min":0.625,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1200000000000001,"5min":1.367,"15min":1.367},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.871,"15min":0.872},"min":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.4199999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.68200000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640304,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319456,"kb_used_data":18004,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94052384,"statfs":{"total":96636764160,"available":96309641216,"internally_reserved":0,"allocated":18436096,"data_stored":17477987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85899999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60599999999999998},"min":{"1min":0.36299999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.871,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.56899999999999995}]},{"osd":1,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.83899999999999997,"15min":0.85299999999999998},"min":{"1min":0.64500000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.117,"5min":1.147,"15min":1.147},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.873,"15min":0.90500000000000003},"min":{"1min":0.64200000000000002,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.98799999999999999,"5min":1.77,"15min":1.77},"last":0.81899999999999995}]},{"osd":4,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.85599999999999998,"15min":0.85699999999999998},"min":{"1min":0.70799999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.88400000000000001,"15min":0.89600000000000002},"min":{"1min":0.76500000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":0.77300000000000002}]},{"osd":5,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91900000000000004,"15min":0.95499999999999996},"min":{"1min":0.623,"5min":0.62,"15min":0.62},"max":{"1min":1.123,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.91800000000000004,"15min":0.92600000000000005},"min":{"1min":0.77500000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.72099999999999997}]}]},{"osd":1,"up_from":66,"seq":283467841560,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344696,"kb_used_data":46700,"kb_used_omap":4,"kb_used_meta":297979,"kb_avail":94027144,"statfs":{"total":96636764160,"available":96283795456,"internally_reserved":0,"allocated":47820800,"data_stored":46838444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305131073},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.499},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":0.59499999999999997}]},{"osd":2,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":0.57999999999999996}]},{"osd":3,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.69799999999999995}]},{"osd":4,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.99099999999999999}]},{"osd":5,"last update":"Wed Apr 24 14:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.745},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.94699999999999995}]}]},{"osd":0,"up_from":11,"seq":47244640303,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348716,"kb_used_data":43164,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94023124,"statfs":{"total":96636764160,"available":96279678976,"internally_reserved":0,"allocated":44199936,"data_stored":43234297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 14:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64800000000000002}]},{"osd":2,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.316,"15min":0.316},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68000000000000005,"15min":0.67900000000000005},"min":{"1min":0.45800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.68899999999999995}]},{"osd":3,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.022,"5min":1.472,"15min":1.472},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81100000000000005,"15min":0.80700000000000005},"min":{"1min":0.51600000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1299999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.76700000000000002}]},{"osd":4,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.89400000000000002,"15min":0.91700000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0309999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.91800000000000004,"15min":0.91900000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2210000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.92200000000000004}]},{"osd":5,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.70299999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.256,"5min":1.641,"15min":1.641},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.92200000000000004,"15min":0.92800000000000005},"min":{"1min":0.66900000000000004,"5min":0.628,"15min":0.628},"max":{"1min":1.1359999999999999,"5min":1.534,"15min":1.534},"last":0.88800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46137730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16785408,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25190400,"data_stored":25166132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:35:53.613 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-24T14:32:21.251204+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=32, tm_sec=21, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=35, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T14:35:53.613 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-24T14:32:27.283594+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=32, tm_sec=27, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=35, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T14:35:53.613 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-24T14:32:34.325783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=32, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=35, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T14:35:53.613 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-24T14:32:27.283594+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=32, tm_sec=27, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=35, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T14:35:53.613 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-24T14:32:34.325783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=32, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=35, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T14:35:53.613 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-24T14:32:27.283594+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=32, tm_sec=27, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=35, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T14:35:53.613 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-24T14:32:34.325783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=32, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=35, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T14:35:53.613 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-24T14:32:27.283594+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=32, tm_sec=27, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=35, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T14:35:53.613 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-24T14:32:34.325783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=32, tm_sec=34, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=35, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T14:35:53.614 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-24T14:32:27.283594+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=32, tm_sec=27, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=35, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T14:35:53.614 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-24T14:32:27.283594+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=32, tm_sec=27, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=35, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T14:35:53.614 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-24T14:32:27.283594+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=32, tm_sec=27, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=35, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T14:35:53.614 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-24T14:32:27.283594+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=32, tm_sec=27, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=14, tm_min=35, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T14:35:53.614 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T14:36:13.616 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T14:36:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-04-24T14:36:13.916 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-04-24T14:36:13.927 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-04-24T14:36:13.234437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80282741,"num_objects":89,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":46,"num_read":8344,"num_read_kb":182682,"num_write":1123,"num_write_kb":273318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":487769303,"num_keys_recovered":143,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":269,"num_omap_keys":28,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1356,"ondisk_log_size":1356,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1975904,"kb_used_data":166576,"kb_used_omap":12,"kb_used_meta":1809267,"kb_avail":564255136,"statfs":{"total":579820584960,"available":577797259264,"internally_reserved":0,"allocated":170573824,"data_stored":164769652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13063,"internal_metadata":1852689657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":43,"num_omap_keys":8,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003368"},"pg_stats":[{"pgid":"1.0","version":"138'119","reported_seq":294,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:55.316725+0000","last_change":"2024-04-24T14:35:55.316658+0000","last_active":"2024-04-24T14:35:55.316725+0000","last_peered":"2024-04-24T14:35:55.316725+0000","last_clean":"2024-04-24T14:35:55.316725+0000","last_became_active":"2024-04-24T14:32:22.271722+0000","last_became_peered":"2024-04-24T14:32:22.271722+0000","last_unstale":"2024-04-24T14:35:55.316725+0000","last_undegraded":"2024-04-24T14:35:55.316725+0000","last_fullsized":"2024-04-24T14:35:55.316725+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'119","last_scrub_stamp":"2024-04-24T14:35:55.316413+0000","last_deep_scrub":"138'119","last_deep_scrub_stamp":"2024-04-24T14:35:55.316413+0000","last_clean_scrub_stamp":"2024-04-24T14:35:55.316413+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:19:23.147052+0000","scrub_duration":0.0079285989999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"138'6","reported_seq":82,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:36:05.604101+0000","last_change":"2024-04-24T14:36:05.604041+0000","last_active":"2024-04-24T14:36:05.604101+0000","last_peered":"2024-04-24T14:36:05.604101+0000","last_clean":"2024-04-24T14:36:05.604101+0000","last_became_active":"2024-04-24T14:35:04.316770+0000","last_became_peered":"2024-04-24T14:35:04.316770+0000","last_unstale":"2024-04-24T14:36:05.604101+0000","last_undegraded":"2024-04-24T14:36:05.604101+0000","last_fullsized":"2024-04-24T14:36:05.604101+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'6","last_scrub_stamp":"2024-04-24T14:36:05.603916+0000","last_deep_scrub":"138'6","last_deep_scrub_stamp":"2024-04-24T14:36:05.603916+0000","last_clean_scrub_stamp":"2024-04-24T14:36:05.603916+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:52:30.768879+0000","scrub_duration":0.030694493999999999,"objects_trimmed":0,"snaptrim_duration":0.00058253000000000005,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"138'202","reported_seq":643,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:53.286415+0000","last_change":"2024-04-24T14:35:53.286384+0000","last_active":"2024-04-24T14:35:53.286415+0000","last_peered":"2024-04-24T14:35:53.286415+0000","last_clean":"2024-04-24T14:35:53.286415+0000","last_became_active":"2024-04-24T14:35:40.002805+0000","last_became_peered":"2024-04-24T14:35:40.002805+0000","last_unstale":"2024-04-24T14:35:53.286415+0000","last_undegraded":"2024-04-24T14:35:53.286415+0000","last_fullsized":"2024-04-24T14:35:53.286415+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'202","last_scrub_stamp":"2024-04-24T14:35:53.286266+0000","last_deep_scrub":"138'202","last_deep_scrub_stamp":"2024-04-24T14:35:53.286266+0000","last_clean_scrub_stamp":"2024-04-24T14:35:53.286266+0000","objects_scrubbed":15,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:36:29.653880+0000","scrub_duration":0.022556448,"objects_trimmed":0,"snaptrim_duration":0.0013315029999999999,"stat_sum":{"num_bytes":19,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":14,"num_read":1693,"num_read_kb":48628,"num_write":128,"num_write_kb":32867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":104988999,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"138'11","reported_seq":242,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:56.349864+0000","last_change":"2024-04-24T14:35:56.349804+0000","last_active":"2024-04-24T14:35:56.349864+0000","last_peered":"2024-04-24T14:35:56.349864+0000","last_clean":"2024-04-24T14:35:56.349864+0000","last_became_active":"2024-04-24T14:35:27.881143+0000","last_became_peered":"2024-04-24T14:35:27.881143+0000","last_unstale":"2024-04-24T14:35:56.349864+0000","last_undegraded":"2024-04-24T14:35:56.349864+0000","last_fullsized":"2024-04-24T14:35:56.349864+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'11","last_scrub_stamp":"2024-04-24T14:35:56.349578+0000","last_deep_scrub":"138'11","last_deep_scrub_stamp":"2024-04-24T14:35:56.349578+0000","last_clean_scrub_stamp":"2024-04-24T14:35:56.349578+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:02:06.080698+0000","scrub_duration":0.0091172500000000004,"objects_trimmed":0,"snaptrim_duration":0.00076176800000000002,"stat_sum":{"num_bytes":78,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":8,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":25165902,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":43,"num_omap_keys":1,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"138'464","reported_seq":3095,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:52.873415+0000","last_change":"2024-04-24T14:35:52.873391+0000","last_active":"2024-04-24T14:35:52.873415+0000","last_peered":"2024-04-24T14:35:52.873415+0000","last_clean":"2024-04-24T14:35:52.873415+0000","last_became_active":"2024-04-24T14:35:29.910973+0000","last_became_peered":"2024-04-24T14:35:29.910973+0000","last_unstale":"2024-04-24T14:35:52.873415+0000","last_undegraded":"2024-04-24T14:35:52.873415+0000","last_fullsized":"2024-04-24T14:35:52.873415+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'464","last_scrub_stamp":"2024-04-24T14:35:52.873289+0000","last_deep_scrub":"138'464","last_deep_scrub_stamp":"2024-04-24T14:35:52.873289+0000","last_clean_scrub_stamp":"2024-04-24T14:35:52.873289+0000","objects_scrubbed":11,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:59:52.802880+0000","scrub_duration":0.018278348999999999,"objects_trimmed":1,"snaptrim_duration":0.0051668080000000002,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":11,"num_read":4739,"num_read_kb":77733,"num_write":290,"num_write_kb":56447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":25165862,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"138'11","reported_seq":250,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:36:00.295766+0000","last_change":"2024-04-24T14:36:00.295716+0000","last_active":"2024-04-24T14:36:00.295766+0000","last_peered":"2024-04-24T14:36:00.295766+0000","last_clean":"2024-04-24T14:36:00.295766+0000","last_became_active":"2024-04-24T14:35:27.884423+0000","last_became_peered":"2024-04-24T14:35:27.884423+0000","last_unstale":"2024-04-24T14:36:00.295766+0000","last_undegraded":"2024-04-24T14:36:00.295766+0000","last_fullsized":"2024-04-24T14:36:00.295766+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'11","last_scrub_stamp":"2024-04-24T14:36:00.295527+0000","last_deep_scrub":"138'11","last_deep_scrub_stamp":"2024-04-24T14:36:00.295527+0000","last_clean_scrub_stamp":"2024-04-24T14:36:00.295527+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:31:23.404361+0000","scrub_duration":0.016989015,"objects_trimmed":0,"snaptrim_duration":0.00085674600000000003,"stat_sum":{"num_bytes":4194380,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":6,"num_write_kb":8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":20971710,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"138'309","reported_seq":853,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:55.295790+0000","last_change":"2024-04-24T14:35:55.295733+0000","last_active":"2024-04-24T14:35:55.295790+0000","last_peered":"2024-04-24T14:35:55.295790+0000","last_clean":"2024-04-24T14:35:55.295790+0000","last_became_active":"2024-04-24T14:35:40.006335+0000","last_became_peered":"2024-04-24T14:35:40.006335+0000","last_unstale":"2024-04-24T14:35:55.295790+0000","last_undegraded":"2024-04-24T14:35:55.295790+0000","last_fullsized":"2024-04-24T14:35:55.295790+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'309","last_scrub_stamp":"2024-04-24T14:35:55.295518+0000","last_deep_scrub":"138'309","last_deep_scrub_stamp":"2024-04-24T14:35:55.295518+0000","last_clean_scrub_stamp":"2024-04-24T14:35:55.295518+0000","objects_scrubbed":13,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:16:21.328177+0000","scrub_duration":0.018846288999999999,"objects_trimmed":0,"snaptrim_duration":0.0014897840000000001,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1035,"num_read_kb":38768,"num_write":359,"num_write_kb":34059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":43040844,"num_keys_recovered":6,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"138'14","reported_seq":120,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:36:03.655986+0000","last_change":"2024-04-24T14:36:03.655926+0000","last_active":"2024-04-24T14:36:03.655986+0000","last_peered":"2024-04-24T14:36:03.655986+0000","last_clean":"2024-04-24T14:36:03.655986+0000","last_became_active":"2024-04-24T14:35:27.885728+0000","last_became_peered":"2024-04-24T14:35:27.885728+0000","last_unstale":"2024-04-24T14:36:03.655986+0000","last_undegraded":"2024-04-24T14:36:03.655986+0000","last_fullsized":"2024-04-24T14:36:03.655986+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'14","last_scrub_stamp":"2024-04-24T14:36:03.655793+0000","last_deep_scrub":"138'14","last_deep_scrub_stamp":"2024-04-24T14:36:03.655793+0000","last_clean_scrub_stamp":"2024-04-24T14:36:03.655793+0000","objects_scrubbed":6,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:16:59.054501+0000","scrub_duration":0.041252427000000001,"objects_trimmed":0,"snaptrim_duration":0.00086492199999999996,"stat_sum":{"num_bytes":16777294,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":14,"num_write_kb":20488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":100663374,"num_keys_recovered":21,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":42,"num_omap_keys":7,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"138'158","reported_seq":293,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:54.290235+0000","last_change":"2024-04-24T14:35:54.290187+0000","last_active":"2024-04-24T14:35:54.290235+0000","last_peered":"2024-04-24T14:35:54.290235+0000","last_clean":"2024-04-24T14:35:54.290235+0000","last_became_active":"2024-04-24T14:34:06.852598+0000","last_became_peered":"2024-04-24T14:34:06.852598+0000","last_unstale":"2024-04-24T14:35:54.290235+0000","last_undegraded":"2024-04-24T14:35:54.290235+0000","last_fullsized":"2024-04-24T14:35:54.290235+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'158","last_scrub_stamp":"2024-04-24T14:35:54.290078+0000","last_deep_scrub":"138'158","last_deep_scrub_stamp":"2024-04-24T14:35:54.290078+0000","last_clean_scrub_stamp":"2024-04-24T14:35:54.290078+0000","objects_scrubbed":11,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:08:45.422654+0000","scrub_duration":0.016870534,"objects_trimmed":0,"snaptrim_duration":0.00154912,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":10,"num_read":707,"num_read_kb":17181,"num_write":118,"num_write_kb":16433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":16777362,"num_keys_recovered":17,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"138'26","reported_seq":68,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:58.331836+0000","last_change":"2024-04-24T14:35:58.331810+0000","last_active":"2024-04-24T14:35:58.331836+0000","last_peered":"2024-04-24T14:35:58.331836+0000","last_clean":"2024-04-24T14:35:58.331836+0000","last_became_active":"2024-04-24T14:35:40.001994+0000","last_became_peered":"2024-04-24T14:35:40.001994+0000","last_unstale":"2024-04-24T14:35:58.331836+0000","last_undegraded":"2024-04-24T14:35:58.331836+0000","last_fullsized":"2024-04-24T14:35:58.331836+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'26","last_scrub_stamp":"2024-04-24T14:35:58.331699+0000","last_deep_scrub":"138'26","last_deep_scrub_stamp":"2024-04-24T14:35:58.331699+0000","last_clean_scrub_stamp":"2024-04-24T14:35:58.331699+0000","objects_scrubbed":9,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:31:36.184867+0000","scrub_duration":0.060267389999999997,"objects_trimmed":0,"snaptrim_duration":0.00061472399999999998,"stat_sum":{"num_bytes":20971636,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":16,"num_read_kb":135,"num_write":17,"num_write_kb":36868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":58720410,"num_keys_recovered":6,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":35,"num_omap_keys":3,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"138'14","reported_seq":193,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:58.367559+0000","last_change":"2024-04-24T14:35:58.367494+0000","last_active":"2024-04-24T14:35:58.367559+0000","last_peered":"2024-04-24T14:35:58.367559+0000","last_clean":"2024-04-24T14:35:58.367559+0000","last_became_active":"2024-04-24T14:32:28.321312+0000","last_became_peered":"2024-04-24T14:32:28.321312+0000","last_unstale":"2024-04-24T14:35:58.367559+0000","last_undegraded":"2024-04-24T14:35:58.367559+0000","last_fullsized":"2024-04-24T14:35:58.367559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'14","last_scrub_stamp":"2024-04-24T14:35:58.367275+0000","last_deep_scrub":"138'14","last_deep_scrub_stamp":"2024-04-24T14:35:58.367275+0000","last_clean_scrub_stamp":"2024-04-24T14:35:58.367275+0000","objects_scrubbed":6,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:25:49.109104+0000","scrub_duration":0.048979380000000003,"objects_trimmed":0,"snaptrim_duration":0.001227351,"stat_sum":{"num_bytes":20971598,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":128,"num_write":6,"num_write_kb":20481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"138'10","reported_seq":163,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:36:00.234086+0000","last_change":"2024-04-24T14:36:00.234023+0000","last_active":"2024-04-24T14:36:00.234086+0000","last_peered":"2024-04-24T14:36:00.234086+0000","last_clean":"2024-04-24T14:36:00.234086+0000","last_became_active":"2024-04-24T14:35:40.000089+0000","last_became_peered":"2024-04-24T14:35:40.000089+0000","last_unstale":"2024-04-24T14:36:00.234086+0000","last_undegraded":"2024-04-24T14:36:00.234086+0000","last_fullsized":"2024-04-24T14:36:00.234086+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'10","last_scrub_stamp":"2024-04-24T14:36:00.233782+0000","last_deep_scrub":"138'10","last_deep_scrub_stamp":"2024-04-24T14:36:00.233782+0000","last_clean_scrub_stamp":"2024-04-24T14:36:00.233782+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:38:03.590792+0000","scrub_duration":0.016184133,"objects_trimmed":0,"snaptrim_duration":0.0013719380000000001,"stat_sum":{"num_bytes":4194342,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":25165862,"num_keys_recovered":2,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1,"num_omap_keys":1,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"138'12","reported_seq":226,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-24T14:35:59.309531+0000","last_change":"2024-04-24T14:35:59.309483+0000","last_active":"2024-04-24T14:35:59.309531+0000","last_peered":"2024-04-24T14:35:59.309531+0000","last_clean":"2024-04-24T14:35:59.309531+0000","last_became_active":"2024-04-24T14:34:06.852864+0000","last_became_peered":"2024-04-24T14:34:06.852864+0000","last_unstale":"2024-04-24T14:35:59.309531+0000","last_undegraded":"2024-04-24T14:35:59.309531+0000","last_fullsized":"2024-04-24T14:35:59.309531+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'12","last_scrub_stamp":"2024-04-24T14:35:59.309318+0000","last_deep_scrub":"138'12","last_deep_scrub_stamp":"2024-04-24T14:35:59.309318+0000","last_clean_scrub_stamp":"2024-04-24T14:35:59.309318+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:59:33.838244+0000","scrub_duration":0.012330839999999999,"objects_trimmed":0,"snaptrim_duration":0.000833036,"stat_sum":{"num_bytes":114,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":8,"num_write_kb":8196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16777330,"num_keys_recovered":16,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":74,"num_omap_keys":8,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":19,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":46,"num_read":8174,"num_read_kb":182310,"num_write":895,"num_write_kb":139806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":189973067,"num_keys_recovered":73,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1133,"ondisk_log_size":1133,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":79692354,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":54,"num_read_kb":272,"num_write":69,"num_write_kb":131096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":297796236,"num_keys_recovered":70,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":269,"num_omap_keys":28,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159473664,"data_stored":159384708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640308,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":331740,"kb_used_data":30292,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94040100,"statfs":{"total":96636764160,"available":96297062400,"internally_reserved":0,"allocated":31019008,"data_stored":30060920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79600000000000004,"15min":0.81299999999999994},"min":{"1min":0.61299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0529999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.80100000000000005,"15min":0.80800000000000005},"min":{"1min":0.54800000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.23,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Apr 24 14:36:03 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.80400000000000005,"15min":0.79400000000000004},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.0009999999999999,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.79800000000000004,"15min":0.80000000000000004},"min":{"1min":0.54900000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.82399999999999995}]},{"osd":2,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.83799999999999997},"min":{"1min":0.54100000000000004,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.026,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.83899999999999997,"15min":0.85599999999999998},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0860000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.95599999999999996}]},{"osd":3,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79200000000000004,"15min":0.82099999999999995},"min":{"1min":0.52000000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.133,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.82299999999999995,"15min":0.85899999999999999},"min":{"1min":0.55800000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.99299999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.54600000000000004}]},{"osd":4,"last update":"Wed Apr 24 14:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.86099999999999999,"15min":0.873},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1259999999999999,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.93799999999999994,"15min":0.97899999999999998},"min":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.103,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.91000000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640308,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303072,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94068768,"statfs":{"total":96636764160,"available":96326418432,"internally_reserved":0,"allocated":1662976,"data_stored":700809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.78100000000000003,"15min":0.78000000000000003},"min":{"1min":0.49399999999999999,"5min":0.379,"15min":0.379},"max":{"1min":1.0800000000000001,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78500000000000003,"15min":0.80100000000000005},"min":{"1min":0.60499999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0049999999999999,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.89300000000000002}]},{"osd":1,"last update":"Wed Apr 24 14:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85199999999999998,"15min":0.85799999999999998},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.034,"5min":1.204,"15min":1.204},"last":1.02},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.79800000000000004,"15min":0.79600000000000004},"min":{"1min":0.61699999999999999,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.80500000000000005}]},{"osd":2,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.83499999999999996,"15min":0.86099999999999999},"min":{"1min":0.63700000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.0189999999999999,"5min":1.113,"15min":1.113},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.64000000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0589999999999999,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.95399999999999996}]},{"osd":3,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83099999999999996,"15min":0.85099999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1970000000000001,"5min":1.931,"15min":1.931},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.85699999999999998,"15min":0.873},"min":{"1min":0.61499999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.161,"5min":1.591,"15min":1.591},"last":0.754}]},{"osd":5,"last update":"Wed Apr 24 14:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.79300000000000004,"15min":0.78700000000000003},"min":{"1min":0.57999999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0629999999999999,"5min":1.087,"15min":1.087},"last":0.63},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.90800000000000003,"15min":0.92700000000000005},"min":{"1min":0.57999999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.68000000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640306,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328240,"kb_used_data":26792,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94043600,"statfs":{"total":96636764160,"available":96300646400,"internally_reserved":0,"allocated":27435008,"data_stored":26457195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.753,"15min":0.77300000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0009999999999999,"5min":1.282,"15min":1.282},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80300000000000005,"15min":0.80100000000000005},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":1.048}]},{"osd":1,"last update":"Wed Apr 24 14:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.82999999999999996,"15min":0.82399999999999995},"min":{"1min":0.63,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.80000000000000004,"15min":0.80800000000000005},"min":{"1min":0.58599999999999997,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0549999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.70299999999999996}]},{"osd":2,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84999999999999998,"15min":0.872},"min":{"1min":0.64900000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.006,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.91900000000000004,"15min":0.93600000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.238,"5min":1.337,"15min":1.337},"last":0.79600000000000004}]},{"osd":4,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.72899999999999998,"15min":0.72599999999999998},"min":{"1min":0.63200000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.52000000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.83199999999999996}]},{"osd":5,"last update":"Wed Apr 24 14:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79400000000000004,"15min":0.78800000000000003},"min":{"1min":0.625,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1200000000000001,"5min":1.367,"15min":1.367},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.871,"15min":0.872},"min":{"1min":0.52500000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.4199999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.159}]}]},{"osd":2,"up_from":11,"seq":47244640309,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":319452,"kb_used_data":18004,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94052388,"statfs":{"total":96636764160,"available":96309645312,"internally_reserved":0,"allocated":18436096,"data_stored":17477987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85899999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.60599999999999998},"min":{"1min":0.36299999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.871,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.66200000000000003}]},{"osd":1,"last update":"Wed Apr 24 14:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.66900000000000004,"15min":0.68100000000000005},"min":{"1min":0.39500000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.877,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.64400000000000002,"15min":0.64000000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.998}]},{"osd":3,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.83899999999999997,"15min":0.85299999999999998},"min":{"1min":0.64500000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.117,"5min":1.147,"15min":1.147},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.873,"15min":0.90500000000000003},"min":{"1min":0.64200000000000002,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.98799999999999999,"5min":1.77,"15min":1.77},"last":1.161}]},{"osd":4,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.85599999999999998,"15min":0.85699999999999998},"min":{"1min":0.70799999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.1060000000000001},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.88400000000000001,"15min":0.89600000000000002},"min":{"1min":0.76500000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":0.95299999999999996}]},{"osd":5,"last update":"Wed Apr 24 14:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91900000000000004,"15min":0.95499999999999996},"min":{"1min":0.623,"5min":0.62,"15min":0.62},"max":{"1min":1.123,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.0229999999999999,"5min":0.91800000000000004,"15min":0.92600000000000005},"min":{"1min":0.77500000000000002,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.0569999999999999}]}]},{"osd":1,"up_from":66,"seq":283467841565,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":344692,"kb_used_data":46700,"kb_used_omap":4,"kb_used_meta":297979,"kb_avail":94027148,"statfs":{"total":96636764160,"available":96283799552,"internally_reserved":0,"allocated":47820800,"data_stored":46838444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305131073},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 14:36:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.55600000000000005,"15min":0.55800000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.872,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59799999999999998,"15min":0.59499999999999997},"min":{"1min":0.375,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.80500000000000005,"5min":2.032,"15min":2.032},"last":0.65900000000000003}]},{"osd":2,"last update":"Wed Apr 24 14:36:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.41499999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.995,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.67000000000000004,"15min":0.65000000000000002},"min":{"1min":0.47499999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.9180000000000001,"5min":2.9180000000000001,"15min":2.9180000000000001},"last":0.62}]},{"osd":3,"last update":"Wed Apr 24 14:36:05 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.69599999999999995,"5min":0.377,"15min":0.377},"max":{"1min":1.089,"5min":2.141,"15min":2.141},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.89900000000000002,"15min":0.88800000000000001},"min":{"1min":0.69799999999999995,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":3.0550000000000002,"5min":3.0550000000000002,"15min":3.0550000000000002},"last":0.89000000000000001}]},{"osd":4,"last update":"Wed Apr 24 14:36:05 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.89000000000000001,"15min":0.89200000000000002},"min":{"1min":0.68899999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.1679999999999999,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":1.006},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"min":{"1min":0.76200000000000001,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.1120000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.74199999999999999}]},{"osd":5,"last update":"Wed Apr 24 14:36:05 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91800000000000004,"15min":0.91900000000000004},"min":{"1min":0.745,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.3009999999999999,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.92600000000000005,"15min":0.92800000000000005},"min":{"1min":0.63900000000000001,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.095,"5min":2.335,"15min":2.335},"last":0.81799999999999995}]}]},{"osd":0,"up_from":11,"seq":47244640307,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":348708,"kb_used_data":43164,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94023132,"statfs":{"total":96636764160,"available":96279687168,"internally_reserved":0,"allocated":44199936,"data_stored":43234297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 14:36:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.36399999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85299999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.65300000000000002}]},{"osd":2,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.316,"15min":0.316},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.68000000000000005,"15min":0.67900000000000005},"min":{"1min":0.45800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.60899999999999999}]},{"osd":3,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.022,"5min":1.472,"15min":1.472},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81100000000000005,"15min":0.80700000000000005},"min":{"1min":0.51600000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1299999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.82999999999999996}]},{"osd":4,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.89400000000000002,"15min":0.91700000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0309999999999999,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.91800000000000004,"15min":0.91900000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2210000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.92500000000000004}]},{"osd":5,"last update":"Wed Apr 24 14:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.70299999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.256,"5min":1.641,"15min":1.641},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.92200000000000004,"15min":0.92800000000000005},"min":{"1min":0.66900000000000004,"5min":0.628,"15min":0.628},"max":{"1min":1.1359999999999999,"5min":1.534,"15min":1.534},"last":0.86599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46137730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16785408,"data_stored":16777292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25190400,"data_stored":25166132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29364224,"data_stored":29360206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T14:36:13.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-24T14:36:14.241 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-24T14:36:14.241 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-24T14:36:14.242 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-24T14:36:14.242 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T14:36:14.306 INFO:tasks.ceph.osd.0:Stopped 2024-04-24T14:36:14.306 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-24T14:36:14.306 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T14:36:14.383 INFO:tasks.ceph.osd.1:Stopped 2024-04-24T14:36:14.383 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-24T14:36:14.383 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T14:36:14.462 INFO:tasks.ceph.osd.2:Stopped 2024-04-24T14:36:14.463 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-24T14:36:14.463 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T14:36:14.525 INFO:tasks.ceph.osd.3:Stopped 2024-04-24T14:36:14.526 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-24T14:36:14.526 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T14:36:14.608 INFO:tasks.ceph.osd.4:Stopped 2024-04-24T14:36:14.608 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-24T14:36:14.608 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T14:36:14.672 INFO:tasks.ceph.osd.5:Stopped 2024-04-24T14:36:14.672 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-24T14:36:14.672 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-24T14:36:14.672 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T14:36:14.698 INFO:tasks.ceph.mgr.x:Stopped 2024-04-24T14:36:14.698 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-24T14:36:14.698 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-24T14:36:14.698 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T14:36:14.716 INFO:tasks.ceph.mon.a:Stopped 2024-04-24T14:36:14.717 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-24T14:36:14.717 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T14:36:14.731 INFO:tasks.ceph.mon.c:Stopped 2024-04-24T14:36:14.732 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-24T14:36:14.732 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T14:36:14.751 INFO:tasks.ceph.mon.b:Stopped 2024-04-24T14:36:14.751 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-24T14:36:14.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | head -n 1 2024-04-24T14:36:14.771 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi090.front.sepia.ceph.com 2024-04-24T14:36:14.771 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-24T14:36:15.165 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi090.front.sepia.ceph.com 2024-04-24T14:36:15.165 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-24T14:36:15.574 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi090.front.sepia.ceph.com 2024-04-24T14:36:15.574 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-24T14:36:15.814 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T14:36:15.814 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-24T14:36:16.034 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T14:36:16.034 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-24T14:36:16.247 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T14:36:16.247 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-24T14:36:16.446 INFO:tasks.ceph:Archiving mon data... 2024-04-24T14:36:16.447 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510/data/mon.a.tgz 2024-04-24T14:36:16.447 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-04-24T14:36:16.454 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.jz9CGivDVy 2024-04-24T14:36:16.454 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.jz9CGivDVy 2024-04-24T14:36:16.544 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.jz9CGivDVy 2024-04-24T14:36:16.562 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.jz9CGivDVy is 314KB 2024-04-24T14:36:16.623 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.jz9CGivDVy 2024-04-24T14:36:16.630 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510/data/mon.c.tgz 2024-04-24T14:36:16.630 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-04-24T14:36:16.637 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.0FRd5ZWR8d 2024-04-24T14:36:16.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.0FRd5ZWR8d 2024-04-24T14:36:16.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.0FRd5ZWR8d 2024-04-24T14:36:16.816 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.0FRd5ZWR8d is 439KB 2024-04-24T14:36:16.839 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.0FRd5ZWR8d 2024-04-24T14:36:16.845 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510/data/mon.b.tgz 2024-04-24T14:36:16.845 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-04-24T14:36:16.852 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.bK3q9DGTlz 2024-04-24T14:36:16.852 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.bK3q9DGTlz 2024-04-24T14:36:17.007 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.bK3q9DGTlz 2024-04-24T14:36:17.030 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.bK3q9DGTlz is 439KB 2024-04-24T14:36:17.097 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.bK3q9DGTlz 2024-04-24T14:36:17.104 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-24T14:36:17.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T14:36:17.106 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T14:36:17.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T14:36:17.125 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-24T14:36:17.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-24T14:36:17.128 DEBUG:teuthology.orchestra.run.smithi103:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-24T14:36:17.153 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-24T14:36:17.164 DEBUG:teuthology.orchestra.run.smithi173:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-24T14:36:17.186 INFO:teuthology.orchestra.run.smithi173.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-24T14:36:17.189 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-24T14:36:17.197 INFO:tasks.ceph:Archiving crash dumps... 2024-04-24T14:36:17.199 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510/remote/smithi090/crash 2024-04-24T14:36:17.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T14:36:17.218 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510/remote/smithi103/crash 2024-04-24T14:36:17.219 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T14:36:17.235 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510/remote/smithi173/crash 2024-04-24T14:36:17.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T14:36:17.251 INFO:tasks.ceph:Compressing logs... 2024-04-24T14:36:17.252 DEBUG:teuthology.orchestra.run.smithi090:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T14:36:17.254 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T14:36:17.256 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T14:36:17.276 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26501.log 2024-04-24T14:36:17.277 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40002.log 2024-04-24T14:36:17.277 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.26501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32741.log 2024-04-24T14:36:17.277 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26501.log.gz 2024-04-24T14:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35835.log 2024-04-24T14:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40002.log.gz 2024-04-24T14:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33794.log 2024-04-24T14:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32741.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.32741.log.gz--verbose 2024-04-24T14:36:17.279 INFO:teuthology.orchestra.run.smithi090.stderr:-- /var/log/ceph/ceph-client.admin.36202.log 2024-04-24T14:36:17.280 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35835.log: /var/log/ceph/ceph-client.admin.33794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33794.log.gz 2024-04-24T14:36:17.280 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35835.log.gz 2024-04-24T14:36:17.280 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37261.log 2024-04-24T14:36:17.280 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38438.log 2024-04-24T14:36:17.281 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20173.log 2024-04-24T14:36:17.281 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-24T14:36:17.283 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36202.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.36202.log.gz-5 2024-04-24T14:36:17.283 INFO:teuthology.orchestra.run.smithi090.stderr:--verbose -- /var/log/ceph/ceph-client.admin.36878.log 2024-04-24T14:36:17.283 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37261.log.gz 2024-04-24T14:36:17.283 INFO:teuthology.orchestra.run.smithi090.stderr:gzip /var/log/ceph/ceph-client.admin.38438.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33634.log 2024-04-24T14:36:17.283 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38438.log.gz 2024-04-24T14:36:17.283 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40446.log 2024-04-24T14:36:17.283 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36878.log.gz 2024-04-24T14:36:17.283 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40170.log 2024-04-24T14:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33634.log.gz 2024-04-24T14:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40455.log 2024-04-24T14:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40446.log.gz 2024-04-24T14:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40170.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.27523.log 2024-04-24T14:36:17.284 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40170.log.gz 2024-04-24T14:36:17.295 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20282.log 2024-04-24T14:36:17.295 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.20173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20173.log.gz 2024-04-24T14:36:17.295 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20229.log 2024-04-24T14:36:17.295 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.20282.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-24T14:36:17.295 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20282.log.gz 2024-04-24T14:36:17.295 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-24T14:36:17.295 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.20229.log: /var/log/ceph/ceph-osd.5.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20229.log.gz 2024-04-24T14:36:17.295 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20146.log 2024-04-24T14:36:17.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-24T14:36:17.296 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20254.log 2024-04-24T14:36:17.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.20146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20201.log 2024-04-24T14:36:17.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.audit.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20146.log.gz 2024-04-24T14:36:17.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.20254.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-24T14:36:17.296 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20254.log.gz 2024-04-24T14:36:17.296 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-24T14:36:17.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.20201.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-24T14:36:17.296 INFO:teuthology.orchestra.run.smithi103.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20201.log.gz 2024-04-24T14:36:17.296 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-osd.3.log: 87.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-24T14:36:17.296 INFO:teuthology.orchestra.run.smithi103.stderr: 84.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-24T14:36:17.375 INFO:teuthology.orchestra.run.smithi103.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-24T14:36:17.422 INFO:teuthology.orchestra.run.smithi103.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-24T14:36:17.479 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37643.log 2024-04-24T14:36:17.480 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40455.log.gz 2024-04-24T14:36:17.480 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36109.log 2024-04-24T14:36:17.480 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/objectstore_tool.27523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24996.log 2024-04-24T14:36:17.480 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37643.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-24T14:36:17.480 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37643.log.gz 2024-04-24T14:36:17.480 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36109.log: /var/log/ceph/ceph-client.admin.24996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36109.log.gz 2024-04-24T14:36:17.480 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36574.log 2024-04-24T14:36:17.480 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24996.log.gz 2024-04-24T14:36:17.480 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24810.log 2024-04-24T14:36:17.480 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25021.log 2024-04-24T14:36:17.480 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36574.log.gz 2024-04-24T14:36:17.480 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.24810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39414.log 2024-04-24T14:36:17.480 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24810.log.gz 2024-04-24T14:36:17.480 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25021.log.gz 2024-04-24T14:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40935.log 2024-04-24T14:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25872.log 2024-04-24T14:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39414.log.gz 2024-04-24T14:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38251.log 2024-04-24T14:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40935.log.gz 2024-04-24T14:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41223.log 2024-04-24T14:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25872.log.gz 2024-04-24T14:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41736.log 2024-04-24T14:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38251.log.gz 2024-04-24T14:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41223.log.gz 2024-04-24T14:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41439.log 2024-04-24T14:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41385.log 0.0% 2024-04-24T14:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41736.log.gz 2024-04-24T14:36:17.481 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36385.log 2024-04-24T14:36:17.482 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41439.log.gz 2024-04-24T14:36:17.482 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41385.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41385.log.gz 2024-04-24T14:36:17.482 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25595.log 2024-04-24T14:36:17.482 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36385.log: 95.2% -- replaced with /var/log/ceph/objectstore_tool.27523.log.gz 2024-04-24T14:36:17.482 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37456.log 2024-04-24T14:36:17.482 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36385.log.gz 2024-04-24T14:36:17.482 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39881.log 2024-04-24T14:36:17.482 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25595.log.gz 2024-04-24T14:36:17.482 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41761.log 2024-04-24T14:36:17.482 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37456.log.gz 2024-04-24T14:36:17.482 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32527.log 2024-04-24T14:36:17.482 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39881.log: /var/log/ceph/ceph-client.admin.41761.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39881.log.gz 2024-04-24T14:36:17.482 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41761.log.gz 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34250.log 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37560.log 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32527.log: /var/log/ceph/ceph-client.admin.34250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32527.log.gz 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34250.log.gz 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33456.log 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40610.log 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37560.log: /var/log/ceph/ceph-client.admin.33456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24756.log 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37560.log.gz 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33456.log.gz 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36082.log 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40610.log.gz 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24849.log 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.24756.log: /var/log/ceph/ceph-client.admin.36082.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36082.log.gz -- replaced with /var/log/ceph/ceph-client.admin.24756.log.gz 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-24T14:36:17.483 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33054.log 2024-04-24T14:36:17.484 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34165.log /var/log/ceph/ceph-client.admin.24849.log: 2024-04-24T14:36:17.484 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24849.log.gz 2024-04-24T14:36:17.484 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33054.log.gz 2024-04-24T14:36:17.484 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33147.log 2024-04-24T14:36:17.484 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41277.log 2024-04-24T14:36:17.484 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34165.log.gz 2024-04-24T14:36:17.484 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33147.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33147.log.gz 2024-04-24T14:36:17.484 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25222.log 2024-04-24T14:36:17.484 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41277.log.gz 2024-04-24T14:36:17.484 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40116.log 2024-04-24T14:36:17.484 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25222.log.gz 2024-04-24T14:36:17.484 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37286.log 2024-04-24T14:36:17.484 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24853.log 2024-04-24T14:36:17.485 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40116.log.gz 2024-04-24T14:36:17.485 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40030.log 2024-04-24T14:36:17.485 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37286.log.gz 2024-04-24T14:36:17.485 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.24853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24853.log.gz 2024-04-24T14:36:17.485 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25541.log 2024-04-24T14:36:17.485 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39458.log 2024-04-24T14:36:17.485 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25541.log: /var/log/ceph/ceph-client.admin.41358.log 2024-04-24T14:36:17.485 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40030.log.gz 2024-04-24T14:36:17.485 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25541.log.gz 2024-04-24T14:36:17.485 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39458.log.gz 2024-04-24T14:36:17.485 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34073.log 2024-04-24T14:36:17.485 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41358.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool..log 2024-04-24T14:36:17.485 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41358.log.gz 2024-04-24T14:36:17.485 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24857.log 2024-04-24T14:36:17.486 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34073.log.gz 2024-04-24T14:36:17.486 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/objectstore_tool..log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25568.log 2024-04-24T14:36:17.486 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.24857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24857.log.gz 2024-04-24T14:36:17.486 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36600.log 2024-04-24T14:36:17.486 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41493.log 2024-04-24T14:36:17.486 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40359.log 2024-04-24T14:36:17.486 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25568.log.gz 2024-04-24T14:36:17.486 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36600.log.gz 2024-04-24T14:36:17.486 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40459.log 2024-04-24T14:36:17.486 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41493.log.gz 2024-04-24T14:36:17.486 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40359.log.gz 2024-04-24T14:36:17.486 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33970.log 2024-04-24T14:36:17.486 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40089.log 2024-04-24T14:36:17.487 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40459.log.gz 2024-04-24T14:36:17.487 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41547.log 2024-04-24T14:36:17.487 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33970.log.gz 2024-04-24T14:36:17.487 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25367.log 2024-04-24T14:36:17.487 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40089.log.gz 2024-04-24T14:36:17.487 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40824.log 2024-04-24T14:36:17.487 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41547.log.gz 2024-04-24T14:36:17.487 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25512.log 2024-04-24T14:36:17.487 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25367.log.gz 2024-04-24T14:36:17.487 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29954.log 2024-04-24T14:36:17.487 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40824.log.gz 2024-04-24T14:36:17.487 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38405.log 2024-04-24T14:36:17.487 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25512.log.gz 2024-04-24T14:36:17.487 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38737.log 2024-04-24T14:36:17.488 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.29954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29954.log.gz 2024-04-24T14:36:17.488 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41169.log 2024-04-24T14:36:17.488 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38405.log.gz 2024-04-24T14:36:17.488 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36969.log 2024-04-24T14:36:17.488 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38737.log.gz 2024-04-24T14:36:17.488 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41169.log: /var/log/ceph/ceph-client.admin.24729.log 2024-04-24T14:36:17.488 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41169.log.gz 2024-04-24T14:36:17.488 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35570.log 2024-04-24T14:36:17.488 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36969.log.gz 2024-04-24T14:36:17.488 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25178.log 2024-04-24T14:36:17.488 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.24729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24729.log.gz 2024-04-24T14:36:17.488 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25381.log 2024-04-24T14:36:17.488 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35570.log.gz 2024-04-24T14:36:17.489 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20307.log 2024-04-24T14:36:17.489 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25178.log.gz 2024-04-24T14:36:17.489 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37151.log 2024-04-24T14:36:17.489 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25381.log.gz 2024-04-24T14:36:17.489 INFO:teuthology.orchestra.run.smithi090.stderr:gzip /var/log/ceph/ceph-client.admin.20307.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.25698.log 2024-04-24T14:36:17.489 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20307.log.gz 2024-04-24T14:36:17.489 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39949.log 2024-04-24T14:36:17.489 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37151.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37151.log.gz 2024-04-24T14:36:17.489 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 /var/log/ceph/ceph-client.admin.25698.log: --verbose -- /var/log/ceph/ceph-client.admin.25024.log 2024-04-24T14:36:17.489 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25698.log.gz 2024-04-24T14:36:17.489 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39388.log 2024-04-24T14:36:17.489 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39949.log.gz 2024-04-24T14:36:17.489 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25161.log 2024-04-24T14:36:17.489 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25024.log.gz 2024-04-24T14:36:17.490 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29981.log 2024-04-24T14:36:17.490 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39388.log.gz 2024-04-24T14:36:17.490 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25161.log: /var/log/ceph/ceph-client.admin.40386.log 2024-04-24T14:36:17.490 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25161.log.gz 2024-04-24T14:36:17.490 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26570.log 2024-04-24T14:36:17.490 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.29981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29981.log.gz 2024-04-24T14:36:17.490 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35698.log 2024-04-24T14:36:17.490 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40386.log.gz 2024-04-24T14:36:17.490 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38898.log 2024-04-24T14:36:17.490 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.26570.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26570.log.gz 2024-04-24T14:36:17.490 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20189.log 2024-04-24T14:36:17.490 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35698.log.gz 2024-04-24T14:36:17.490 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25354.log 2024-04-24T14:36:17.490 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38898.log.gz 2024-04-24T14:36:17.491 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.20189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24619.log 2024-04-24T14:36:17.491 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20189.log.gz 2024-04-24T14:36:17.491 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37903.log 2024-04-24T14:36:17.491 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25354.log.gz 2024-04-24T14:36:17.491 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25941.log 2024-04-24T14:36:17.491 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.24619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24619.log.gz 2024-04-24T14:36:17.491 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33523.log 2024-04-24T14:36:17.491 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37903.log.gz 2024-04-24T14:36:17.491 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40970.log 2024-04-24T14:36:17.491 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25941.log: 95.5% -- replaced with /var/log/ceph/objectstore_tool..log.gz 2024-04-24T14:36:17.491 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25941.log.gz 2024-04-24T14:36:17.491 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39056.log 2024-04-24T14:36:17.491 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33523.log.gz 2024-04-24T14:36:17.491 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37766.log 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40970.log.gz 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33280.log 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39056.log.gz 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40623.log 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37766.log: /var/log/ceph/ceph-client.admin.33280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37766.log.gz 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24855.log 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33280.log.gz 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr:gzip /var/log/ceph/ceph-client.admin.40623.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32694.log 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40623.log.gz 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.24855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41466.log 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24855.log.gz 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38784.log 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32694.log.gz 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39096.log 2024-04-24T14:36:17.492 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41466.log.gz 2024-04-24T14:36:17.493 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38784.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-24T14:36:17.493 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38784.log.gz 2024-04-24T14:36:17.493 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-24T14:36:17.493 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39096.log.gz 2024-04-24T14:36:17.493 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26391.log 2024-04-24T14:36:17.493 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26051.log 2024-04-24T14:36:17.493 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.26391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38475.log 2024-04-24T14:36:17.493 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26391.log.gz 2024-04-24T14:36:17.493 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26015.log 2024-04-24T14:36:17.493 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.26051.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26051.log.gz 2024-04-24T14:36:17.493 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41331.log 2024-04-24T14:36:17.493 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38475.log.gz 2024-04-24T14:36:17.493 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.26015.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26015.log.gz-5 2024-04-24T14:36:17.493 INFO:teuthology.orchestra.run.smithi090.stderr:--verbose -- /var/log/ceph/ceph-client.admin.32985.log 2024-04-24T14:36:17.494 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40847.log 2024-04-24T14:36:17.494 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41331.log.gz 2024-04-24T14:36:17.494 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32474.log 2024-04-24T14:36:17.494 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32985.log.gz 2024-04-24T14:36:17.494 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40847.log: gzip -5 0.0%--verbose -- replaced with /var/log/ceph/ceph-client.admin.40847.log.gz-- 2024-04-24T14:36:17.494 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25622.log 2024-04-24T14:36:17.494 INFO:teuthology.orchestra.run.smithi090.stderr:gzip /var/log/ceph/ceph-client.admin.32474.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40413.log 2024-04-24T14:36:17.494 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32474.log.gz 2024-04-24T14:36:17.494 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26156.log 2024-04-24T14:36:17.494 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25622.log.gz 2024-04-24T14:36:17.494 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40772.log 2024-04-24T14:36:17.494 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40413.log.gz 2024-04-24T14:36:17.494 INFO:teuthology.orchestra.run.smithi090.stderr: 93.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-24T14:36:17.495 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36821.log 2024-04-24T14:36:17.495 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.26156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26156.log.gz 2024-04-24T14:36:17.495 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41682.log 2024-04-24T14:36:17.495 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40772.log.gz/var/log/ceph/ceph-client.admin.36821.log: 2024-04-24T14:36:17.495 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26460.log 2024-04-24T14:36:17.495 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36821.log.gz 2024-04-24T14:36:17.495 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37067.log 2024-04-24T14:36:17.495 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41682.log.gz 2024-04-24T14:36:17.495 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25173.log 2024-04-24T14:36:17.495 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.26460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26460.log.gz 2024-04-24T14:36:17.495 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33384.log 2024-04-24T14:36:17.495 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37067.log.gz 2024-04-24T14:36:17.495 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26191.log 2024-04-24T14:36:17.495 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25173.log.gz 2024-04-24T14:36:17.496 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38343.log 2024-04-24T14:36:17.496 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33384.log.gz 2024-04-24T14:36:17.496 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40062.log 2024-04-24T14:36:17.496 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.26191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26191.log.gz 2024-04-24T14:36:17.496 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.38343.log: -- /var/log/ceph/ceph-client.admin.41142.log 2024-04-24T14:36:17.496 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38343.log.gz 2024-04-24T14:36:17.496 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35880.log 2024-04-24T14:36:17.496 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40062.log.gz 2024-04-24T14:36:17.496 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38788.log 2024-04-24T14:36:17.496 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41142.log.gz 2024-04-24T14:36:17.496 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32593.log 2024-04-24T14:36:17.496 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35880.log.gz 2024-04-24T14:36:17.496 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37155.log 2024-04-24T14:36:17.496 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38788.log.gz 2024-04-24T14:36:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41709.log 2024-04-24T14:36:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32593.log.gz 2024-04-24T14:36:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30214.log 2024-04-24T14:36:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37155.log.gz 2024-04-24T14:36:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36198.log 2024-04-24T14:36:17.497 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41709.log.gz 2024-04-24T14:36:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40765.log 2024-04-24T14:36:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.30214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30214.log.gz 2024-04-24T14:36:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40224.log 2024-04-24T14:36:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36198.log.gz 2024-04-24T14:36:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25654.log 2024-04-24T14:36:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40765.log.gz 2024-04-24T14:36:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39976.log 2024-04-24T14:36:17.497 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40224.log.gz 2024-04-24T14:36:17.498 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39541.log 2024-04-24T14:36:17.498 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25654.log.gz 2024-04-24T14:36:17.498 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35539.log 2024-04-24T14:36:17.498 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39976.log.gz 2024-04-24T14:36:17.498 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25323.log 2024-04-24T14:36:17.498 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39541.log.gz 2024-04-24T14:36:17.498 INFO:teuthology.orchestra.run.smithi090.stderr:gzip /var/log/ceph/ceph-client.admin.35539.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38644.log 2024-04-24T14:36:17.498 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35539.log.gz 2024-04-24T14:36:17.498 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35959.log 2024-04-24T14:36:17.498 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25323.log.gz 2024-04-24T14:36:17.498 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38290.log 2024-04-24T14:36:17.498 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38644.log.gz 2024-04-24T14:36:17.498 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34130.log 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35959.log.gz 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31215.log 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38290.log.gz 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39578.log 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34130.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34130.log.gz 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40626.log 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.31215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31215.log.gz 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr:gzip /var/log/ceph/ceph-client.admin.39578.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.25003.log 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39578.log.gz 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25170.log 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40626.log.gz 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34316.log 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37036.log 0.0% 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25003.log.gz 2024-04-24T14:36:17.499 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25170.log.gz 2024-04-24T14:36:17.500 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26277.log 2024-04-24T14:36:17.500 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34316.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34316.log.gz 2024-04-24T14:36:17.500 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34576.log 2024-04-24T14:36:17.500 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37036.log.gz 2024-04-24T14:36:17.500 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36748.log 2024-04-24T14:36:17.500 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.26277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26277.log.gz 2024-04-24T14:36:17.500 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40454.log 2024-04-24T14:36:17.500 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34576.log.gz 2024-04-24T14:36:17.500 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39459.log 2024-04-24T14:36:17.500 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36748.log.gz 2024-04-24T14:36:17.500 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25905.log 2024-04-24T14:36:17.500 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40454.log.gz 2024-04-24T14:36:17.500 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26125.log 2024-04-24T14:36:17.500 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39459.log.gz 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24676.log 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25905.log.gz 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40930.log 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.26125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26125.log.gz 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41196.log 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.24676.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24676.log.gz 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr:gzip /var/log/ceph/ceph-client.admin.40930.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38178.log 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40930.log.gz 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27090.log 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41196.log.gz 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38178.log.gz 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.16576.log 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.27090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27090.log.gz 2024-04-24T14:36:17.501 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41115.log 2024-04-24T14:36:17.502 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph.tmp-client.admin.16576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24783.log 2024-04-24T14:36:17.502 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.16576.log.gz 2024-04-24T14:36:17.502 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37728.log 2024-04-24T14:36:17.502 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41115.log.gz 2024-04-24T14:36:17.502 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33483.log 2024-04-24T14:36:17.502 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.24783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24783.log.gz 2024-04-24T14:36:17.502 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39167.log 2024-04-24T14:36:17.502 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37728.log.gz 2024-04-24T14:36:17.502 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38133.log 2024-04-24T14:36:17.502 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33483.log.gz 2024-04-24T14:36:17.502 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38070.log 2024-04-24T14:36:17.502 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39167.log.gz 2024-04-24T14:36:17.502 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40768.log 2024-04-24T14:36:17.503 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38133.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36303.log 2024-04-24T14:36:17.503 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38133.log.gz 2024-04-24T14:36:17.503 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38070.log.gz 2024-04-24T14:36:17.503 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33559.log 2024-04-24T14:36:17.503 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40768.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40768.log.gz-5 2024-04-24T14:36:17.503 INFO:teuthology.orchestra.run.smithi090.stderr:--verbose -- /var/log/ceph/ceph-client.admin.29138.log 2024-04-24T14:36:17.503 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36303.log.gz 2024-04-24T14:36:17.503 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36931.log 2024-04-24T14:36:17.503 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41304.log 2024-04-24T14:36:17.503 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33559.log.gz 2024-04-24T14:36:17.503 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.29138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29138.log.gz 2024-04-24T14:36:17.503 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40603.log 2024-04-24T14:36:17.503 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36931.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36931.log.gz 2024-04-24T14:36:17.503 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40197.log 2024-04-24T14:36:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41304.log.gz 2024-04-24T14:36:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24845.log 2024-04-24T14:36:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40603.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40603.log.gz 2024-04-24T14:36:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27094.log 2024-04-24T14:36:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40197.log.gz 2024-04-24T14:36:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25327.log 2024-04-24T14:36:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.24845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24845.log.gz 2024-04-24T14:36:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39795.log 2024-04-24T14:36:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.27094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27094.log.gz 2024-04-24T14:36:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.28330.log 2024-04-24T14:36:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25327.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25327.log.gz 2024-04-24T14:36:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34436.log 2024-04-24T14:36:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39795.log.gz 2024-04-24T14:36:17.504 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26615.log 2024-04-24T14:36:17.505 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/objectstore_tool.28330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40927.log 2024-04-24T14:36:17.505 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34436.log.gz 2024-04-24T14:36:17.505 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.26615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32434.log 2024-04-24T14:36:17.505 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26615.log.gz 2024-04-24T14:36:17.505 INFO:teuthology.orchestra.run.smithi090.stderr:gzip /var/log/ceph/ceph-client.admin.40927.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.25165.log 2024-04-24T14:36:17.505 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40927.log.gz 2024-04-24T14:36:17.505 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36463.log 2024-04-24T14:36:17.505 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32434.log.gz 2024-04-24T14:36:17.505 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32928.log 2024-04-24T14:36:17.505 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25165.log.gz 2024-04-24T14:36:17.505 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39003.log 2024-04-24T14:36:17.505 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36463.log.gz 2024-04-24T14:36:17.505 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33597.log 2024-04-24T14:36:17.505 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32928.log.gz 2024-04-24T14:36:17.506 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33080.log 0.0% 2024-04-24T14:36:17.506 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39003.log.gz 2024-04-24T14:36:17.506 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 /var/log/ceph/ceph-client.admin.33597.log: --verbose -- /var/log/ceph/ceph-client.admin.36014.log 2024-04-24T14:36:17.506 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33597.log.gz 2024-04-24T14:36:17.506 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41010.log 2024-04-24T14:36:17.506 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33080.log.gz 2024-04-24T14:36:17.506 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25011.log 2024-04-24T14:36:17.506 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36014.log.gz 2024-04-24T14:36:17.506 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41010.log.gz 2024-04-24T14:36:17.506 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41250.log 2024-04-24T14:36:17.506 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32809.log 2024-04-24T14:36:17.506 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25011.log.gz 2024-04-24T14:36:17.506 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34218.log 2024-04-24T14:36:17.506 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41250.log.gz 2024-04-24T14:36:17.507 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33333.log 2024-04-24T14:36:17.507 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32809.log.gz 2024-04-24T14:36:17.507 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40305.log 2024-04-24T14:36:17.507 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34218.log: gzip /var/log/ceph/ceph-client.admin.33333.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33671.log 2024-04-24T14:36:17.507 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34218.log.gz 2024-04-24T14:36:17.507 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33333.log.gz 2024-04-24T14:36:17.507 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36350.log 2024-04-24T14:36:17.507 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40305.log.gz 2024-04-24T14:36:17.507 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39194.log 2024-04-24T14:36:17.507 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33671.log: 95.4% -- replaced with /var/log/ceph/objectstore_tool.28330.log.gz 2024-04-24T14:36:17.507 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33671.log.gz 2024-04-24T14:36:17.507 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36491.log 2024-04-24T14:36:17.507 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36350.log.gz 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40251.log 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39194.log.gz 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40984.log 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36491.log.gz 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41520.log 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40251.log.gz 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr:gzip /var/log/ceph/ceph-client.admin.40984.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39303.log 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40984.log.gz 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37618.log 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41520.log.gz 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr:gzip /var/log/ceph/ceph-client.admin.39303.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35458.log 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39303.log.gz 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28538.log 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37618.log.gz 2024-04-24T14:36:17.508 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37507.log 2024-04-24T14:36:17.509 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35458.log.gz 2024-04-24T14:36:17.509 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35645.log 2024-04-24T14:36:17.509 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.28538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28538.log.gz 2024-04-24T14:36:17.509 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41412.log 2024-04-24T14:36:17.509 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37507.log.gz 2024-04-24T14:36:17.509 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39644.log 2024-04-24T14:36:17.509 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35645.log.gz 2024-04-24T14:36:17.509 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32959.log 2024-04-24T14:36:17.509 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41412.log.gz 2024-04-24T14:36:17.509 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28296.log 2024-04-24T14:36:17.509 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39644.log.gz 2024-04-24T14:36:17.509 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41088.log 2024-04-24T14:36:17.509 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26314.log 2024-04-24T14:36:17.509 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32959.log.gz 2024-04-24T14:36:17.510 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.28296.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28296.log.gz 2024-04-24T14:36:17.510 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36426.log 2024-04-24T14:36:17.510 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41088.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41088.log.gz-5 2024-04-24T14:36:17.510 INFO:teuthology.orchestra.run.smithi090.stderr:--verbose -- /var/log/ceph/ceph-client.admin.25792.log 2024-04-24T14:36:17.510 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.26314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26314.log.gz 2024-04-24T14:36:17.510 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40278.log 2024-04-24T14:36:17.510 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36426.log.gz 2024-04-24T14:36:17.510 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36273.log 2024-04-24T14:36:17.510 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25792.log.gz 2024-04-24T14:36:17.510 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37329.log 2024-04-24T14:36:17.510 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40278.log.gz 2024-04-24T14:36:17.510 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39770.log 2024-04-24T14:36:17.510 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36273.log.gz 2024-04-24T14:36:17.510 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39617.log 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37329.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37329.log.gz 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40460.log 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39770.log: /var/log/ceph/ceph-client.admin.39617.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.39770.log.gz-5 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr:--verbose -- /var/log/ceph/ceph-client.admin.36710.log 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39617.log.gz 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32850.log 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40460.log.gz 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37975.log 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36710.log.gz 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.26679.log 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32850.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32850.log.gz 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25697.log 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37975.log.gz 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/objectstore_tool.26679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31435.log 2024-04-24T14:36:17.511 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38370.log 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25697.log.gz 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.31435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31435.log.gz 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34507.log 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38370.log.gz 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33239.log 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40604.log 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34507.log.gz 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39827.log 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33239.log.gz 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40604.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40604.log.gz 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39031.log 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39827.log.gz 2024-04-24T14:36:17.512 INFO:teuthology.orchestra.run.smithi090.stderr: 91.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-24T14:36:17.513 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.0.log: 95.4% -- replaced with /var/log/ceph/objectstore_tool.26679.log.gz 2024-04-24T14:36:17.513 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41628.log 2024-04-24T14:36:17.513 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25834.log 2024-04-24T14:36:17.513 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39031.log.gz 2024-04-24T14:36:17.513 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38962.log 2024-04-24T14:36:17.513 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41628.log.gz 2024-04-24T14:36:17.513 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35781.log 2024-04-24T14:36:17.513 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25834.log.gz 2024-04-24T14:36:17.513 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24702.log 2024-04-24T14:36:17.513 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38962.log.gz 2024-04-24T14:36:17.513 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37398.log 2024-04-24T14:36:17.513 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35781.log.gz 2024-04-24T14:36:17.513 INFO:teuthology.orchestra.run.smithi090.stderr:gzip /var/log/ceph/ceph-client.admin.24702.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35954.log 2024-04-24T14:36:17.514 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24702.log.gz 2024-04-24T14:36:17.514 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.29133.log 2024-04-24T14:36:17.514 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37398.log.gz 2024-04-24T14:36:17.514 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36683.log 2024-04-24T14:36:17.514 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35954.log.gz 2024-04-24T14:36:17.514 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/objectstore_tool.29133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25485.log 2024-04-24T14:36:17.514 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20395.log 2024-04-24T14:36:17.514 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36683.log.gz 2024-04-24T14:36:17.514 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25485.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25485.log.gz 2024-04-24T14:36:17.514 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33728.log 2024-04-24T14:36:17.514 INFO:teuthology.orchestra.run.smithi090.stderr:gzip /var/log/ceph/ceph-client.admin.20395.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41655.log 2024-04-24T14:36:17.514 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20395.log.gz 2024-04-24T14:36:17.514 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33728.log.gz 2024-04-24T14:36:17.514 INFO:teuthology.orchestra.run.smithi090.stderr: 95.4% -- replaced with /var/log/ceph/objectstore_tool.29133.log.gz 2024-04-24T14:36:17.515 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32619.log 2024-04-24T14:36:17.515 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32890.log 2024-04-24T14:36:17.515 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41655.log.gz 2024-04-24T14:36:17.515 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39689.log 2024-04-24T14:36:17.515 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32619.log.gz 2024-04-24T14:36:17.515 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34997.log 2024-04-24T14:36:17.515 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32890.log.gz 2024-04-24T14:36:17.515 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25010.log 2024-04-24T14:36:17.515 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39689.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34345.log 2024-04-24T14:36:17.515 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39689.log.gz/var/log/ceph/ceph-client.admin.34997.log: 2024-04-24T14:36:17.515 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37876.log 2024-04-24T14:36:17.515 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34997.log.gz 2024-04-24T14:36:17.515 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25010.log.gz 2024-04-24T14:36:17.515 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38566.log 2024-04-24T14:36:17.516 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34345.log.gz 2024-04-24T14:36:17.516 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33887.log 2024-04-24T14:36:17.516 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37876.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37876.log.gz 2024-04-24T14:36:17.516 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37828.log 2024-04-24T14:36:17.516 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38566.log.gz 2024-04-24T14:36:17.516 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40332.log 2024-04-24T14:36:17.516 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33887.log.gz 2024-04-24T14:36:17.516 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33830.log 2024-04-24T14:36:17.516 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37828.log.gz 2024-04-24T14:36:17.516 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25458.log 2024-04-24T14:36:17.516 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40332.log.gz 2024-04-24T14:36:17.516 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40143.log 2024-04-24T14:36:17.516 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33830.log.gz 2024-04-24T14:36:17.516 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33933.log 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25458.log.gz 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24856.log 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40143.log.gz 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33192.log 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33933.log.gz 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39733.log 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.24856.log: /var/log/ceph/ceph-client.admin.33192.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24856.log.gzgzip 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.38534.log 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33192.log.gz 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39733.log.gz 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41574.log 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35723.log 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38534.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39278.log 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38534.log.gz 2024-04-24T14:36:17.517 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40618.log 2024-04-24T14:36:17.518 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35723.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35723.log.gz 2024-04-24T14:36:17.518 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41574.log.gz 2024-04-24T14:36:17.518 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25694.log 2024-04-24T14:36:17.518 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39278.log.gz 2024-04-24T14:36:17.518 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40618.log.gz 2024-04-24T14:36:17.518 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38687.log 2024-04-24T14:36:17.518 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40456.log 2024-04-24T14:36:17.518 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.25694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25694.log.gz 2024-04-24T14:36:17.518 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34037.log 2024-04-24T14:36:17.518 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38687.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38687.log.gz 2024-04-24T14:36:17.518 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41601.log 2024-04-24T14:36:17.518 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40456.log.gz 2024-04-24T14:36:17.519 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-24T14:36:17.519 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34037.log.gz 2024-04-24T14:36:17.519 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40799.log 2024-04-24T14:36:17.519 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41601.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36138.log 2024-04-24T14:36:17.519 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41601.log.gz 2024-04-24T14:36:17.519 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.40799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38013.log 2024-04-24T14:36:17.519 INFO:teuthology.orchestra.run.smithi090.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40799.log.gz 2024-04-24T14:36:17.519 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36138.log.gz 2024-04-24T14:36:17.519 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38013.log.gz 2024-04-24T14:36:17.519 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19314.log 2024-04-24T14:36:17.520 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17473.log 2024-04-24T14:36:17.520 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18611.log 2024-04-24T14:36:17.520 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19314.log.gz 2024-04-24T14:36:17.520 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16926.log 2024-04-24T14:36:17.520 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17473.log: 52.1% -- replaced with /var/log/ceph/ceph-client.admin.17473.log.gz 2024-04-24T14:36:17.520 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18611.log.gz 2024-04-24T14:36:17.520 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17973.log 2024-04-24T14:36:17.520 INFO:teuthology.orchestra.run.smithi173.stderr:gzip /var/log/ceph/ceph-client.admin.16926.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.17097.log 2024-04-24T14:36:17.520 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16926.log.gz 2024-04-24T14:36:17.520 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20084.log 2024-04-24T14:36:17.520 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17973.log.gz 2024-04-24T14:36:17.520 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17622.log 2024-04-24T14:36:17.520 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17097.log: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.17097.log.gz 2024-04-24T14:36:17.520 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17706.log 2024-04-24T14:36:17.521 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20084.log: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.20084.log.gz 2024-04-24T14:36:17.521 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16698.log 2024-04-24T14:36:17.521 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17622.log: 17.5% -- replaced with /var/log/ceph/ceph-client.admin.17622.log.gz 2024-04-24T14:36:17.521 INFO:teuthology.orchestra.run.smithi173.stderr:gzip /var/log/ceph/ceph-client.admin.17706.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.18786.log 2024-04-24T14:36:17.521 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17706.log.gz 2024-04-24T14:36:17.521 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17500.log 2024-04-24T14:36:17.521 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16698.log.gz 2024-04-24T14:36:17.521 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18918.log 2024-04-24T14:36:17.521 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18786.log: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.18786.log.gz 2024-04-24T14:36:17.521 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17500.log.gzgzip 2024-04-24T14:36:17.521 INFO:teuthology.orchestra.run.smithi173.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.20120.log 2024-04-24T14:36:17.521 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20248.log 2024-04-24T14:36:17.521 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18918.log: /var/log/ceph/ceph-client.admin.20120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18918.log.gz 2024-04-24T14:36:17.521 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20120.log.gz 2024-04-24T14:36:17.522 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18072.log 2024-04-24T14:36:17.522 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20322.log 2024-04-24T14:36:17.522 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20248.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.20248.log.gz--verbose 2024-04-24T14:36:17.522 INFO:teuthology.orchestra.run.smithi173.stderr:-- /var/log/ceph/ceph-client.admin.20534.log 2024-04-24T14:36:17.522 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18072.log.gz 2024-04-24T14:36:17.522 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19357.log 2024-04-24T14:36:17.522 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20322.log.gz 2024-04-24T14:36:17.522 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19540.log 2024-04-24T14:36:17.522 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20534.log: /var/log/ceph/ceph-client.admin.19357.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20534.log.gz 2024-04-24T14:36:17.522 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18795.log 2024-04-24T14:36:17.522 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19357.log.gz 2024-04-24T14:36:17.522 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18234.log 2024-04-24T14:36:17.522 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19540.log.gz 2024-04-24T14:36:17.523 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19651.log 2024-04-24T14:36:17.523 INFO:teuthology.orchestra.run.smithi173.stderr: 25.9% -- replaced with /var/log/ceph/ceph-client.admin.18795.log.gz 2024-04-24T14:36:17.523 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18599.log 2024-04-24T14:36:17.523 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17287.log 2024-04-24T14:36:17.523 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18234.log.gz 2024-04-24T14:36:17.523 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19651.log: 56.3% -- replaced with /var/log/ceph/ceph-client.admin.19651.log.gz 2024-04-24T14:36:17.523 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18599.log.gz 2024-04-24T14:36:17.523 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20023.log 2024-04-24T14:36:17.523 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19885.log 2024-04-24T14:36:17.523 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17287.log.gz 2024-04-24T14:36:17.523 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17665.log 2024-04-24T14:36:17.523 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20023.log.gz 2024-04-24T14:36:17.523 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18045.log 2024-04-24T14:36:17.523 INFO:teuthology.orchestra.run.smithi173.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.19885.log.gz 2024-04-24T14:36:17.524 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20438.log 2024-04-24T14:36:17.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17665.log.gz 2024-04-24T14:36:17.524 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20206.log 2024-04-24T14:36:17.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19642.log 2024-04-24T14:36:17.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20438.log: 34.8% -- replaced with /var/log/ceph/ceph-client.admin.18045.log.gz 2024-04-24T14:36:17.524 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20438.log.gz 2024-04-24T14:36:17.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19370.log 2024-04-24T14:36:17.524 INFO:teuthology.orchestra.run.smithi173.stderr: 34.3% -- replaced with /var/log/ceph/ceph-client.admin.20206.log.gz 2024-04-24T14:36:17.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19642.log: gzip 18.7% -- replaced with /var/log/ceph/ceph-client.admin.19642.log.gz 2024-04-24T14:36:17.524 INFO:teuthology.orchestra.run.smithi173.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.17445.log 2024-04-24T14:36:17.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18658.log 2024-04-24T14:36:17.524 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19370.log.gz 2024-04-24T14:36:17.524 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17082.log 2024-04-24T14:36:17.524 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17445.log: /var/log/ceph/ceph-client.admin.18658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17445.log.gz 2024-04-24T14:36:17.525 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19640.log 2024-04-24T14:36:17.525 INFO:teuthology.orchestra.run.smithi173.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.18658.log.gz 2024-04-24T14:36:17.525 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19322.log 2024-04-24T14:36:17.525 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17082.log.gz 2024-04-24T14:36:17.525 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17348.log 2024-04-24T14:36:17.525 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19513.log 2024-04-24T14:36:17.525 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19640.log.gz 2024-04-24T14:36:17.525 INFO:teuthology.orchestra.run.smithi173.stderr: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.19322.log.gz 2024-04-24T14:36:17.525 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18320.log 2024-04-24T14:36:17.525 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17348.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17348.log.gz 2024-04-24T14:36:17.525 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19513.log.gz 2024-04-24T14:36:17.525 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19389.log 2024-04-24T14:36:17.525 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20245.log 2024-04-24T14:36:17.525 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18320.log.gz 2024-04-24T14:36:17.526 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20459.log 2024-04-24T14:36:17.526 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19389.log: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.19389.log.gz 2024-04-24T14:36:17.526 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17181.log 2024-04-24T14:36:17.526 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20245.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20245.log.gz 2024-04-24T14:36:17.526 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18891.log 2024-04-24T14:36:17.526 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20459.log.gz 2024-04-24T14:36:17.526 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17181.log.gz 2024-04-24T14:36:17.526 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19852.log 2024-04-24T14:36:17.526 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17824.log 2024-04-24T14:36:17.526 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18891.log: gzip 35.1%-5 -- replaced with /var/log/ceph/ceph-client.admin.18891.log.gz--verbose -- 2024-04-24T14:36:17.526 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18146.log 2024-04-24T14:36:17.526 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19852.log: gzip 3.6%-5 -- replaced with /var/log/ceph/ceph-client.admin.19852.log.gz--verbose -- 2024-04-24T14:36:17.526 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17133.log 2024-04-24T14:36:17.527 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17824.log: 17.5% -- replaced with /var/log/ceph/ceph-client.admin.17824.log.gz 2024-04-24T14:36:17.527 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18486.log 2024-04-24T14:36:17.527 INFO:teuthology.orchestra.run.smithi173.stderr: 34.4% -- replaced with /var/log/ceph/ceph-client.admin.18146.log.gz 2024-04-24T14:36:17.527 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18422.log 2024-04-24T14:36:17.527 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17133.log.gz 2024-04-24T14:36:17.527 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17578.log 2024-04-24T14:36:17.527 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18486.log.gz 2024-04-24T14:36:17.527 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18627.log 2024-04-24T14:36:17.527 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18422.log.gz 2024-04-24T14:36:17.527 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16542.log 2024-04-24T14:36:17.527 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17578.log.gz 2024-04-24T14:36:17.527 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20307.log 2024-04-24T14:36:17.527 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18627.log.gz 2024-04-24T14:36:17.527 INFO:teuthology.orchestra.run.smithi173.stderr:gzip /var/log/ceph/ceph-client.admin.16542.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.18822.log 2024-04-24T14:36:17.528 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16542.log.gz 2024-04-24T14:36:17.528 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17833.log 2024-04-24T14:36:17.528 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20307.log.gz 2024-04-24T14:36:17.528 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17897.log 2024-04-24T14:36:17.528 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18822.log.gz 2024-04-24T14:36:17.528 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17106.log 2024-04-24T14:36:17.528 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17833.log: 54.8% -- replaced with /var/log/ceph/ceph-client.admin.17833.log.gz 2024-04-24T14:36:17.528 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16594.log 2024-04-24T14:36:17.528 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17897.log.gz 2024-04-24T14:36:17.528 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18730.log 2024-04-24T14:36:17.528 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17106.log: 52.4% -- replaced with /var/log/ceph/ceph-client.admin.17106.log.gz 2024-04-24T14:36:17.528 INFO:teuthology.orchestra.run.smithi173.stderr:gzip /var/log/ceph/ceph-client.admin.16594.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.19230.log 2024-04-24T14:36:17.528 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16594.log.gz 2024-04-24T14:36:17.528 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18200.log 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18730.log.gz 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20396.log 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19230.log.gz 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19233.log 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18200.log.gz 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17235.log 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20396.log: /var/log/ceph/ceph-client.admin.19233.log: 22.4% -- replaced with /var/log/ceph/ceph-client.admin.20396.log.gz 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.19233.log.gz 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.20417.log 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18946.log 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17235.log.gz 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18608.log 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20417.log.gz 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18849.log 2024-04-24T14:36:17.529 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18946.log: /var/log/ceph/ceph-client.admin.18608.log: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.18946.log.gz 2024-04-24T14:36:17.530 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18608.log.gz 2024-04-24T14:36:17.530 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16998.log 2024-04-24T14:36:17.530 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 /var/log/ceph/ceph-client.admin.18849.log: --verbose -- /var/log/ceph/ceph-client.admin.18831.log 2024-04-24T14:36:17.530 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18849.log.gz 2024-04-24T14:36:17.530 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19251.log 2024-04-24T14:36:17.530 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16998.log: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.16998.log.gz 2024-04-24T14:36:17.530 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19436.log 2024-04-24T14:36:17.530 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18831.log.gz 2024-04-24T14:36:17.530 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16672.log 2024-04-24T14:36:17.530 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19251.log: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.19251.log.gz 2024-04-24T14:36:17.530 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16568.log 2024-04-24T14:36:17.530 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19436.log.gz 2024-04-24T14:36:17.530 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19534.log 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16672.log.gz 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17304.log 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16568.log.gz 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18624.log 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19534.log.gz 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr:gzip /var/log/ceph/ceph-client.admin.17304.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.19785.log 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17304.log.gz 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17040.log 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18624.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18624.log.gz 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19986.log 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.19785.log.gz 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19718.log 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr: 25.7% -- replaced with /var/log/ceph/ceph-client.admin.17040.log.gz 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17262.log 2024-04-24T14:36:17.531 INFO:teuthology.orchestra.run.smithi173.stderr: 6.4% -- replaced with /var/log/ceph/ceph-client.admin.19986.log.gz 2024-04-24T14:36:17.532 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17462.log 2024-04-24T14:36:17.532 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19718.log: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.19718.log.gz 2024-04-24T14:36:17.532 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16881.log 2024-04-24T14:36:17.532 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17262.log: 53.9% -- replaced with /var/log/ceph/ceph-client.admin.17262.log.gz 2024-04-24T14:36:17.532 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17603.log 2024-04-24T14:36:17.532 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17462.log.gz 2024-04-24T14:36:17.532 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18459.log 2024-04-24T14:36:17.532 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16881.log.gz 2024-04-24T14:36:17.532 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17603.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17603.log.gz 2024-04-24T14:36:17.532 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18272.log 2024-04-24T14:36:17.532 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17937.log 2024-04-24T14:36:17.532 INFO:teuthology.orchestra.run.smithi173.stderr: 34.2% -- replaced with /var/log/ceph/ceph-client.admin.18459.log.gz 2024-04-24T14:36:17.532 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17860.log 2024-04-24T14:36:17.532 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18272.log.gz 2024-04-24T14:36:17.533 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17946.log 2024-04-24T14:36:17.533 INFO:teuthology.orchestra.run.smithi173.stderr: 17.5% -- replaced with /var/log/ceph/ceph-client.admin.17937.log.gz 2024-04-24T14:36:17.533 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17750.log 2024-04-24T14:36:17.533 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17685.log 2024-04-24T14:36:17.533 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17860.log.gz 2024-04-24T14:36:17.533 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17946.log: /var/log/ceph/ceph-client.admin.17750.log: 34.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.17946.log.gz-5 2024-04-24T14:36:17.533 INFO:teuthology.orchestra.run.smithi173.stderr:--verbose -- /var/log/ceph/ceph-client.admin.17398.log 2024-04-24T14:36:17.533 INFO:teuthology.orchestra.run.smithi173.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.17750.log.gz 2024-04-24T14:36:17.533 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18448.log 2024-04-24T14:36:17.533 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17685.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17685.log.gz 2024-04-24T14:36:17.533 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18245.log 2024-04-24T14:36:17.533 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17398.log.gz 2024-04-24T14:36:17.533 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19577.log 2024-04-24T14:36:17.533 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18448.log.gz 2024-04-24T14:36:17.534 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19078.log 2024-04-24T14:36:17.534 INFO:teuthology.orchestra.run.smithi173.stderr: 34.5% -- replaced with /var/log/ceph/ceph-client.admin.18245.log.gz 2024-04-24T14:36:17.534 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18036.log 2024-04-24T14:36:17.534 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19577.log.gz 2024-04-24T14:36:17.534 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20264.log 2024-04-24T14:36:17.534 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19078.log.gz 2024-04-24T14:36:17.534 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18495.log 2024-04-24T14:36:17.534 INFO:teuthology.orchestra.run.smithi173.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.18036.log.gz 2024-04-24T14:36:17.534 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20355.log 2024-04-24T14:36:17.534 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20264.log: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.20264.log.gz 2024-04-24T14:36:17.534 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20176.log 2024-04-24T14:36:17.534 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18495.log.gz 2024-04-24T14:36:17.534 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17872.log 2024-04-24T14:36:17.534 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20355.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20355.log.gz 2024-04-24T14:36:17.535 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18667.log 2024-04-24T14:36:17.535 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20176.log.gz 2024-04-24T14:36:17.535 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20262.log 2024-04-24T14:36:17.535 INFO:teuthology.orchestra.run.smithi173.stderr: 42.1% -- replaced with /var/log/ceph/ceph-client.admin.17872.log.gz 2024-04-24T14:36:17.535 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17370.log 2024-04-24T14:36:17.535 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20517.log 2024-04-24T14:36:17.535 INFO:teuthology.orchestra.run.smithi173.stderr: 34.9% -- replaced with /var/log/ceph/ceph-client.admin.18667.log.gz 2024-04-24T14:36:17.535 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18934.log 2024-04-24T14:36:17.535 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20262.log.gz 2024-04-24T14:36:17.535 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17370.log.gz 2024-04-24T14:36:17.535 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16906.log 2024-04-24T14:36:17.535 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20517.log.gz 2024-04-24T14:36:17.535 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17195.log 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18934.log.gz 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18929.log 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16906.log.gz 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16861.log 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17195.log: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.17195.log.gz 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17606.log 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18929.log.gz 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19551.log 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16861.log.gz 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16516.log 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17606.log.gz 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18858.log 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr: 45.2% -- replaced with /var/log/ceph/ceph-client.admin.19551.log.gz 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17230.log 2024-04-24T14:36:17.536 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16516.log.gz 2024-04-24T14:36:17.537 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18322.log 2024-04-24T14:36:17.537 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18858.log.gz 2024-04-24T14:36:17.537 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18384.log 2024-04-24T14:36:17.537 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17230.log.gz 2024-04-24T14:36:17.537 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19380.log 2024-04-24T14:36:17.537 INFO:teuthology.orchestra.run.smithi173.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.18322.log.gz 2024-04-24T14:36:17.537 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19542.log 2024-04-24T14:36:17.537 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18384.log.gz 2024-04-24T14:36:17.537 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17900.log 2024-04-24T14:36:17.537 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19380.log: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.19380.log.gz 2024-04-24T14:36:17.537 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18137.log 2024-04-24T14:36:17.537 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19542.log: 19.3% -- replaced with /var/log/ceph/ceph-client.admin.19542.log.gz 2024-04-24T14:36:17.537 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17900.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.17900.log.gz 2024-04-24T14:36:17.537 INFO:teuthology.orchestra.run.smithi173.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.18988.log 2024-04-24T14:36:17.538 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17290.log 2024-04-24T14:36:17.538 INFO:teuthology.orchestra.run.smithi173.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.18137.log.gz 2024-04-24T14:36:17.538 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17142.log 2024-04-24T14:36:17.538 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18988.log: /var/log/ceph/ceph-client.admin.17290.log: 45.1% -- replaced with /var/log/ceph/ceph-client.admin.18988.log.gz 2024-04-24T14:36:17.538 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20057.log 2024-04-24T14:36:17.538 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17290.log.gz 2024-04-24T14:36:17.538 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18331.log 2024-04-24T14:36:17.538 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17142.log: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.17142.log.gz 2024-04-24T14:36:17.538 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17204.log 2024-04-24T14:36:17.538 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20057.log.gz 2024-04-24T14:36:17.538 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19018.log 2024-04-24T14:36:17.538 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18331.log: /var/log/ceph/ceph-client.admin.17204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20195.log 2024-04-24T14:36:17.538 INFO:teuthology.orchestra.run.smithi173.stderr: 33.7% -- replaced with /var/log/ceph/ceph-client.admin.18331.log.gz 2024-04-24T14:36:17.538 INFO:teuthology.orchestra.run.smithi173.stderr: 52.4% -- replaced with /var/log/ceph/ceph-client.admin.17204.log.gz 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17087.log 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19018.log.gz 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17658.log 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20195.log.gz 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18395.log 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17087.log.gz 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20179.log 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17658.log.gz 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18550.log 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18395.log: 33.7% -- replaced with /var/log/ceph/ceph-client.admin.18395.log.gz 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 /var/log/ceph/ceph-client.admin.20179.log: --verbose -- /var/log/ceph/ceph-client.admin.19331.log 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20179.log.gz 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20152.log 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18550.log.gz 2024-04-24T14:36:17.539 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18571.log 2024-04-24T14:36:17.540 INFO:teuthology.orchestra.run.smithi173.stderr: 45.2% -- replaced with /var/log/ceph/ceph-client.admin.19331.log.gz 2024-04-24T14:36:17.540 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18099.log 2024-04-24T14:36:17.540 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20152.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20152.log.gz 2024-04-24T14:36:17.540 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17822.log 2024-04-24T14:36:17.540 INFO:teuthology.orchestra.run.smithi173.stderr: 34.9% -- replaced with /var/log/ceph/ceph-client.admin.18571.log.gz 2024-04-24T14:36:17.540 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18523.log 2024-04-24T14:36:17.540 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18099.log.gz 2024-04-24T14:36:17.540 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19051.log 2024-04-24T14:36:17.540 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17822.log.gz 2024-04-24T14:36:17.540 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19115.log 2024-04-24T14:36:17.540 INFO:teuthology.orchestra.run.smithi173.stderr: 34.4% -- replaced with /var/log/ceph/ceph-client.admin.18523.log.gz 2024-04-24T14:36:17.540 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19952.log 2024-04-24T14:36:17.540 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19051.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19051.log.gz 2024-04-24T14:36:17.541 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17786.log 2024-04-24T14:36:17.541 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19115.log.gz 2024-04-24T14:36:17.541 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18983.log 2024-04-24T14:36:17.541 INFO:teuthology.orchestra.run.smithi173.stderr: 3.6% -- replaced with /var/log/ceph/ceph-client.admin.19952.log.gz 2024-04-24T14:36:17.541 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19306.log 2024-04-24T14:36:17.541 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17786.log.gz 2024-04-24T14:36:17.541 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19302.log 2024-04-24T14:36:17.541 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18983.log.gz 2024-04-24T14:36:17.541 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19586.log 2024-04-24T14:36:17.541 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19306.log.gz 2024-04-24T14:36:17.541 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20197.log 2024-04-24T14:36:17.541 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19302.log: 0.0%/var/log/ceph/ceph-client.admin.19586.log: -- replaced with /var/log/ceph/ceph-client.admin.19302.log.gz 2024-04-24T14:36:17.541 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17464.log 2024-04-24T14:36:17.541 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19586.log.gz 2024-04-24T14:36:17.542 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18370.log 2024-04-24T14:36:17.542 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20197.log: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.20197.log.gz 2024-04-24T14:36:17.542 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18135.log 2024-04-24T14:36:17.542 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17464.log: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.17464.log.gz 2024-04-24T14:36:17.542 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18000.log 2024-04-24T14:36:17.542 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18370.log.gz 2024-04-24T14:36:17.542 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18236.log 2024-04-24T14:36:17.542 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18135.log.gz 2024-04-24T14:36:17.542 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18109.log 2024-04-24T14:36:17.542 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18000.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18000.log.gz 2024-04-24T14:36:17.542 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18431.log 2024-04-24T14:36:17.542 INFO:teuthology.orchestra.run.smithi173.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.18236.log.gz 2024-04-24T14:36:17.542 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18109.log: /var/log/ceph/ceph-client.admin.20082.log 2024-04-24T14:36:17.542 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18109.log.gz 2024-04-24T14:36:17.543 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19594.log 2024-04-24T14:36:17.543 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18431.log.gz 2024-04-24T14:36:17.543 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17342.log 2024-04-24T14:36:17.543 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20082.log.gz 2024-04-24T14:36:17.543 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18882.log 2024-04-24T14:36:17.543 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19594.log.gz 2024-04-24T14:36:17.543 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16824.log 2024-04-24T14:36:17.543 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17342.log.gz 2024-04-24T14:36:17.543 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17151.log 2024-04-24T14:36:17.543 INFO:teuthology.orchestra.run.smithi173.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.18882.log.gz 2024-04-24T14:36:17.543 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19475.log 2024-04-24T14:36:17.543 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16824.log.gz 2024-04-24T14:36:17.543 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17620.log 2024-04-24T14:36:17.543 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17151.log: 26.9% -- replaced with /var/log/ceph/ceph-client.admin.17151.log.gz 2024-04-24T14:36:17.544 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17448.log 2024-04-24T14:36:17.544 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19475.log.gz 2024-04-24T14:36:17.544 INFO:teuthology.orchestra.run.smithi173.stderr:gzip /var/log/ceph/ceph-client.admin.17620.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.18560.log 2024-04-24T14:36:17.544 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17620.log.gz 2024-04-24T14:36:17.544 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20273.log 2024-04-24T14:36:17.544 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17448.log.gz 2024-04-24T14:36:17.544 INFO:teuthology.orchestra.run.smithi173.stderr:gzip /var/log/ceph/ceph-client.admin.18560.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.17306.log 2024-04-24T14:36:17.544 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18560.log.gz 2024-04-24T14:36:17.544 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18034.log 2024-04-24T14:36:17.544 INFO:teuthology.orchestra.run.smithi173.stderr: 34.6% -- replaced with /var/log/ceph/ceph-client.admin.20273.log.gz 2024-04-24T14:36:17.544 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18562.log 2024-04-24T14:36:17.544 INFO:teuthology.orchestra.run.smithi173.stderr: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.17306.log.gz 2024-04-24T14:36:17.544 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17095.log 2024-04-24T14:36:17.544 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18034.log.gz 2024-04-24T14:36:17.545 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17748.log 2024-04-24T14:36:17.545 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18562.log: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.18562.log.gz 2024-04-24T14:36:17.545 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17420.log 2024-04-24T14:36:17.545 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17095.log.gz 2024-04-24T14:36:17.545 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16981.log 2024-04-24T14:36:17.545 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17748.log.gz 2024-04-24T14:36:17.545 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18284.log 2024-04-24T14:36:17.545 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17420.log.gz 2024-04-24T14:36:17.545 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20479.log 2024-04-24T14:36:17.545 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16981.log.gz 2024-04-24T14:36:17.545 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18284.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18284.log.gz 2024-04-24T14:36:17.545 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18512.log 2024-04-24T14:36:17.545 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17631.log 2024-04-24T14:36:17.545 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20479.log: 55.0% -- replaced with /var/log/ceph/ceph-client.admin.20479.log.gz 2024-04-24T14:36:17.546 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18081.log 2024-04-24T14:36:17.546 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18512.log.gz 2024-04-24T14:36:17.546 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18209.log 2024-04-24T14:36:17.546 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18367.log 2024-04-24T14:36:17.546 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18081.log: 57.1% -- replaced with /var/log/ceph/ceph-client.admin.17631.log.gz 2024-04-24T14:36:17.546 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18081.log.gz 2024-04-24T14:36:17.546 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18694.log 2024-04-24T14:36:17.546 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18209.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18209.log.gz 2024-04-24T14:36:17.546 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17805.log 2024-04-24T14:36:17.546 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18367.log.gz 2024-04-24T14:36:17.546 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19112.log 2024-04-24T14:36:17.546 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18694.log.gz 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19366.log 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17805.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17805.log.gz 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17178.log 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19112.log.gz 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18498.log 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19366.log.gz 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17734.log 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17178.log.gz 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17506.log 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18498.log.gz 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19614.log 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17734.log.gz 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19378.log 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17506.log.gz 2024-04-24T14:36:17.547 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19818.log 2024-04-24T14:36:17.548 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19614.log.gz 2024-04-24T14:36:17.548 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17528.log 2024-04-24T14:36:17.548 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19378.log.gz 2024-04-24T14:36:17.548 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18281.log 2024-04-24T14:36:17.548 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19818.log: /var/log/ceph/ceph-client.admin.17528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19287.log 2024-04-24T14:36:17.548 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17528.log.gz 2024-04-24T14:36:17.548 INFO:teuthology.orchestra.run.smithi173.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.19818.log.gz 2024-04-24T14:36:17.548 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20233.log 2024-04-24T14:36:17.548 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18281.log.gz 2024-04-24T14:36:17.548 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17909.log 2024-04-24T14:36:17.548 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18358.log 2024-04-24T14:36:17.548 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19287.log.gz 2024-04-24T14:36:17.548 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20233.log.gz 2024-04-24T14:36:17.548 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17808.log 2024-04-24T14:36:17.549 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17909.log.gz 2024-04-24T14:36:17.549 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19477.log 2024-04-24T14:36:17.549 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18358.log.gz 2024-04-24T14:36:17.549 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18703.log 2024-04-24T14:36:17.549 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17808.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.18294.log 2024-04-24T14:36:17.549 INFO:teuthology.orchestra.run.smithi173.stderr: -- replaced with /var/log/ceph/ceph-client.admin.17808.log.gz 2024-04-24T14:36:17.549 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19477.log: 19.3% -- replaced with /var/log/ceph/ceph-client.admin.19477.log.gz 2024-04-24T14:36:17.549 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19439.log 2024-04-24T14:36:17.549 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18703.log.gz 2024-04-24T14:36:17.549 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20093.log 2024-04-24T14:36:17.549 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18294.log.gz 2024-04-24T14:36:17.549 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17982.log 2024-04-24T14:36:17.549 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19439.log.gz 2024-04-24T14:36:17.549 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20027.log 2024-04-24T14:36:17.550 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20093.log: 53.4% -- replaced with /var/log/ceph/ceph-client.admin.20093.log.gz 2024-04-24T14:36:17.550 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17556.log 2024-04-24T14:36:17.550 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17982.log.gz 2024-04-24T14:36:17.550 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20376.log 2024-04-24T14:36:17.550 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20027.log: /var/log/ceph/ceph-client.admin.17556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17031.log 2024-04-24T14:36:17.550 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20027.log.gz 2024-04-24T14:36:17.550 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17556.log.gz 2024-04-24T14:36:17.550 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19260.log 2024-04-24T14:36:17.550 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20376.log: 22.2% -- replaced with /var/log/ceph/ceph-client.admin.20376.log.gz 2024-04-24T14:36:17.550 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20500.log 2024-04-24T14:36:17.550 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16952.log 2024-04-24T14:36:17.550 INFO:teuthology.orchestra.run.smithi173.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.17031.log.gz 2024-04-24T14:36:17.550 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19260.log: /var/log/ceph/ceph-client.admin.20500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18880.log 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr: 43.7% -- replaced with /var/log/ceph/ceph-client.admin.19260.log.gz 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20500.log.gz 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17084.log 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr: 26.3% -- replaced with /var/log/ceph/ceph-client.admin.16952.log.gz 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18386.log 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18880.log.gz 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18721.log 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17084.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17084.log.gz 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19486.log 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.18386.log.gz 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18173.log 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18721.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18721.log.gz 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17731.log 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19486.log: 45.5% -- replaced with /var/log/ceph/ceph-client.admin.19486.log.gz 2024-04-24T14:36:17.551 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19249.log 2024-04-24T14:36:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18173.log.gz 2024-04-24T14:36:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16620.log 2024-04-24T14:36:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17731.log.gz 2024-04-24T14:36:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18641.log /var/log/ceph/ceph-client.admin.19249.log: 2024-04-24T14:36:17.552 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19249.log.gz 2024-04-24T14:36:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19522.log 2024-04-24T14:36:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16620.log.gz 2024-04-24T14:36:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18641.log: /var/log/ceph/ceph-client.admin.19919.log 2024-04-24T14:36:17.552 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18641.log.gz 2024-04-24T14:36:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16646.log 2024-04-24T14:36:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19522.log.gz 2024-04-24T14:36:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19320.log 2024-04-24T14:36:17.552 INFO:teuthology.orchestra.run.smithi173.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.19919.log.gz 2024-04-24T14:36:17.552 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20319.log 2024-04-24T14:36:17.553 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16646.log.gz 2024-04-24T14:36:17.553 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16841.log 2024-04-24T14:36:17.553 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19320.log.gz 2024-04-24T14:36:17.553 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20036.log 2024-04-24T14:36:17.553 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20319.log.gz 2024-04-24T14:36:17.553 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18182.log 2024-04-24T14:36:17.553 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16841.log: 22.0% -- replaced with /var/log/ceph/ceph-client.admin.16841.log.gz 2024-04-24T14:36:17.553 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17935.log 2024-04-24T14:36:17.553 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20036.log.gz 2024-04-24T14:36:17.553 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17759.log 2024-04-24T14:36:17.553 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18182.log.gz 2024-04-24T14:36:17.553 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19751.log 2024-04-24T14:36:17.553 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17935.log.gz 2024-04-24T14:36:17.553 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18434.log 2024-04-24T14:36:17.554 INFO:teuthology.orchestra.run.smithi173.stderr: 52.9% -- replaced with /var/log/ceph/ceph-client.admin.17759.log.gz 2024-04-24T14:36:17.554 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20125.log 2024-04-24T14:36:17.554 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19751.log: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.19751.log.gz 2024-04-24T14:36:17.554 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19590.log 2024-04-24T14:36:17.554 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18434.log.gz 2024-04-24T14:36:17.554 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.20125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19681.log 2024-04-24T14:36:17.554 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20125.log.gz 2024-04-24T14:36:17.554 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17315.log 2024-04-24T14:36:17.554 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19590.log.gz 2024-04-24T14:36:17.554 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18514.log 2024-04-24T14:36:17.554 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19681.log.gz 2024-04-24T14:36:17.554 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18944.log 2024-04-24T14:36:17.554 INFO:teuthology.orchestra.run.smithi173.stderr: 52.5% -- replaced with /var/log/ceph/ceph-client.admin.17315.log.gz 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18955.log 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.18514.log.gz 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18747.log 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18944.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18944.log.gz 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19449.log 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr: 60.3% -- replaced with /var/log/ceph/ceph-client.admin.18955.log.gz 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19526.log 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18747.log.gz 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18769.log 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19449.log.gz 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18450.log 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16943.log 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19526.log.gz 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18769.log.gz 2024-04-24T14:36:17.555 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19169.log 2024-04-24T14:36:17.556 INFO:teuthology.orchestra.run.smithi173.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.18450.log.gz 2024-04-24T14:36:17.556 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.16943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.19415.log 2024-04-24T14:36:17.556 INFO:teuthology.orchestra.run.smithi173.stderr: 18.1% -- replaced with /var/log/ceph/ceph-client.admin.16943.log.gz 2024-04-24T14:36:17.556 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.18009.log 2024-04-24T14:36:17.556 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19169.log.gz 2024-04-24T14:36:17.556 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20334.log 2024-04-24T14:36:17.556 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19415.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19415.log.gz 2024-04-24T14:36:17.556 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.18009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17007.log 2024-04-24T14:36:17.556 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.18009.log.gz 2024-04-24T14:36:17.556 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.20334.log: -- /var/log/ceph/ceph-client.admin.19187.log 2024-04-24T14:36:17.556 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20334.log.gz 2024-04-24T14:36:17.556 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.17007.log: 4.8% -- replaced with /var/log/ceph/ceph-client.admin.17007.log.gz 2024-04-24T14:36:17.556 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.19187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.19187.log.gz 2024-04-24T14:36:17.556 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-04-24T14:36:17.557 INFO:teuthology.orchestra.run.smithi173.stderr:real 0m0.123s 2024-04-24T14:36:17.557 INFO:teuthology.orchestra.run.smithi173.stderr:user 0m0.231s 2024-04-24T14:36:17.557 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m0.085s 2024-04-24T14:36:17.557 INFO:teuthology.orchestra.run.smithi090.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-24T14:36:17.557 INFO:teuthology.orchestra.run.smithi103.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-24T14:36:17.565 INFO:teuthology.orchestra.run.smithi103.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-24T14:36:17.566 INFO:teuthology.orchestra.run.smithi090.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-24T14:36:17.611 INFO:teuthology.orchestra.run.smithi103.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-24T14:36:17.613 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-04-24T14:36:17.614 INFO:teuthology.orchestra.run.smithi103.stderr:real 0m0.354s 2024-04-24T14:36:17.614 INFO:teuthology.orchestra.run.smithi103.stderr:user 0m1.014s 2024-04-24T14:36:17.614 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m0.059s 2024-04-24T14:36:17.620 INFO:teuthology.orchestra.run.smithi090.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-24T14:36:17.650 INFO:teuthology.orchestra.run.smithi090.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-24T14:36:17.814 INFO:teuthology.orchestra.run.smithi090.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-24T14:36:17.816 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-04-24T14:36:17.816 INFO:teuthology.orchestra.run.smithi090.stderr:real 0m0.558s 2024-04-24T14:36:17.816 INFO:teuthology.orchestra.run.smithi090.stderr:user 0m1.648s 2024-04-24T14:36:17.816 INFO:teuthology.orchestra.run.smithi090.stderr:sys 0m0.132s 2024-04-24T14:36:17.816 INFO:tasks.ceph:Archiving logs... 2024-04-24T14:36:17.816 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510/remote/smithi090/log 2024-04-24T14:36:17.817 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T14:36:18.301 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510/remote/smithi103/log 2024-04-24T14:36:18.302 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T14:36:18.473 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510/remote/smithi173/log 2024-04-24T14:36:18.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T14:36:18.692 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-24T14:36:18.711 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-24T14:36:18.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T14:36:18.714 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T14:36:18.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T14:36:18.805 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T14:36:18.814 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T14:36:18.814 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T14:36:18.816 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T14:36:18.819 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T14:36:18.837 INFO:teuthology.orchestra.run.smithi090.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T14:36:18.837 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================== 2024-04-24T14:36:18.837 INFO:teuthology.orchestra.run.smithi090.stdout:*hv01.front.sepi 44.4.53.2 2 u 15 64 177 0.104 0.398 0.636 2024-04-24T14:36:18.837 INFO:teuthology.orchestra.run.smithi090.stdout:+hv02.front.sepi 63.231.80.2 3 u 18 64 177 0.101 0.841 0.660 2024-04-24T14:36:18.837 INFO:teuthology.orchestra.run.smithi090.stdout:+hv03.front.sepi 74.6.168.72 3 u 15 64 177 0.108 0.405 0.655 2024-04-24T14:36:18.837 INFO:teuthology.orchestra.run.smithi090.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:36:18.839 INFO:teuthology.orchestra.run.smithi103.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T14:36:18.839 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================== 2024-04-24T14:36:18.839 INFO:teuthology.orchestra.run.smithi103.stdout: hv01.front.sepi 44.4.53.2 2 u 17 64 177 0.104 0.364 0.415 2024-04-24T14:36:18.839 INFO:teuthology.orchestra.run.smithi103.stdout:+hv02.front.sepi 63.231.80.2 3 u 16 64 177 0.054 0.205 0.796 2024-04-24T14:36:18.840 INFO:teuthology.orchestra.run.smithi103.stdout:*hv03.front.sepi 74.6.168.72 3 u 18 64 177 0.094 0.569 0.540 2024-04-24T14:36:18.840 INFO:teuthology.orchestra.run.smithi103.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:36:18.841 INFO:teuthology.orchestra.run.smithi173.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T14:36:18.841 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================== 2024-04-24T14:36:18.841 INFO:teuthology.orchestra.run.smithi173.stdout:+hv01.front.sepi 44.4.53.2 2 u 17 64 177 0.094 -0.100 0.254 2024-04-24T14:36:18.841 INFO:teuthology.orchestra.run.smithi173.stdout:+hv02.front.sepi 63.231.80.2 3 u 16 64 177 0.070 0.154 0.246 2024-04-24T14:36:18.841 INFO:teuthology.orchestra.run.smithi173.stdout:*hv03.front.sepi 74.6.168.72 3 u 14 64 177 0.070 0.020 0.301 2024-04-24T14:36:18.841 INFO:teuthology.orchestra.run.smithi173.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-04-24T14:36:18.842 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T14:36:18.852 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T14:36:18.853 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T14:36:18.873 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T14:36:18.971 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T14:36:18.998 INFO:teuthology.task.internal:Duration was 755.835902 seconds 2024-04-24T14:36:18.998 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T14:36:19.031 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T14:36:19.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T14:36:19.034 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T14:36:19.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T14:36:19.089 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T14:36:19.089 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2024-04-24T14:36:19.089 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T14:36:19.103 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-04-24T14:36:19.103 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T14:36:19.118 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-04-24T14:36:19.118 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T14:36:19.134 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T14:36:19.134 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T14:36:19.136 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T14:36:19.139 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T14:36:19.176 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T14:36:19.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T14:36:19.223 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T14:36:19.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T14:36:19.307 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T14:36:19.318 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T14:36:19.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T14:36:19.334 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T14:36:19.348 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T14:36:19.366 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T14:36:19.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T14:36:19.379 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T14:36:19.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T14:36:19.391 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2024-04-24T14:36:19.393 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-04-24T14:36:19.424 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-04-24T14:36:19.440 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T14:36:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T14:36:19.446 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T14:36:19.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T14:36:19.480 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T14:36:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T14:36:19.486 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T14:36:19.497 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T14:36:19.497 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510/remote/smithi090 2024-04-24T14:36:19.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T14:36:19.524 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510/remote/smithi103 2024-04-24T14:36:19.524 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T14:36:19.549 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671510/remote/smithi173 2024-04-24T14:36:19.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T14:36:19.575 INFO:teuthology.task.internal:Removing archive directory... 2024-04-24T14:36:19.575 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T14:36:19.577 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T14:36:19.588 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T14:36:19.620 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T14:36:19.632 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T14:36:19.632 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T14:36:19.690 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T14:36:19.690 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T14:36:19.693 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T14:36:19.695 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T14:36:19.698 INFO:teuthology.orchestra.run.smithi090.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 14:36 /home/ubuntu/cephtest 2024-04-24T14:36:19.700 INFO:teuthology.orchestra.run.smithi103.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 14:36 /home/ubuntu/cephtest 2024-04-24T14:36:19.702 INFO:teuthology.orchestra.run.smithi173.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 14:36 /home/ubuntu/cephtest 2024-04-24T14:36:19.704 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T14:36:19.724 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T14:36:19.793 INFO:teuthology.run:Summary data: description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} thrashers/cache thrashosds-health workloads/rbd_nbd} duration: 755.8359024524689 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-04-24T14:36:19.793 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T14:36:19.889 INFO:teuthology.run:pass